Abstract
Using a cluster randomized trial design, we evaluated the persistence of effects of a research-based model for scaling up educational interventions. The model was implemented in 42 schools in two city districts serving low-resource communities, randomly assigned to three conditions. In pre-kindergarten, the two experimental interventions were identical, but one included follow-through in the kindergarten and first-grade years, including knowledge of the pre-K intervention and ways to build upon that knowledge using learning trajectories. Students in the experimental group scored significantly higher than control students (g = .51 for those who received follow-through intervention in kindergarten and first grade; g = .28 for non–follow-through), and follow-through students scored significantly higher than non–follow-through students (g = .24).
Keywords
Education needs generalizable models of scale-up and longitudinal research evaluating the persistence of the effect of their implementations (Borman, 2007; Cuban & Usdan, 2003; McDonald, Keesler, Kauffman, & Schneider, 2006), especially in early mathematics (National Mathematics Advisory Panel, 2008). We created a research-based model for scale-up called TRIAD (Technology-enhanced, Research-based, Instruction, Assessment, and professional Development; see Sarama, Clements, Starkey, Klein, & Wakeley, 2008; Sarama, Clements, Wolfe, & Spitler, 2012), with the intent to generalize to both other subject matter areas and other grade levels. We implemented in the early childhood years because long-range benefits to children are greatest for interventions in that period (Carneiro & Heckman, 2003; Clements & Sarama, 2009) and because there is conflicting evidence as to whether the effects of early interventions persist or fade. The present study is an experimental evaluation of this implementation, the third and final year of TRIAD’s follow-through intervention.
The rationale for situating the intervention in mathematics is that children from low-resource communities and who are members of linguistic and ethnic minority groups demonstrate significantly lower levels of mathematics achievement than children from higher-resource, nonminority communities (Denton & West, 2002; Duncan & Magnuson, in press; National Mathematics Advisory Panel, 2008). Such differences are evident from the earliest years (National Research Council, 2001, 2009; Sarama & Clements, 2009). This is important because early mathematics competence predicts later achievement even into high school (National Mathematics Advisory Panel, 2008; National Research Council, 2009; Stevenson & Newman, 1986), with persistent difficulties with mathematics one of the strongest predictors of failure to graduate high school and enter college (Duncan & Magnuson, 2011). Interventions to address these early differences may benefit low-resource and minority children more because they have fewer educational opportunities in their homes and communities (Brooks-Gunn, 2003; Carneiro & Heckman, 2003; Natriello, McDill, & Pallas, 1990; Raudenbush, 2009). Such interventions have been shown to be effective; however, most have not been taken to scale and often used the individual child as the unit of analysis, despite their assignment to treatments by class or school, which can inflate findings (Case, Griffin, & Kelly, 1999; for a review, see Clements & Sarama, 2011; Griffin & Case, 1997; Griffin, Case, & Siegler, 1994; Klein, Starkey, Clements, Sarama, & Iyer, 2008; National Mathematics Advisory Panel, 2008; Starkey, Klein, & Wakeley, 2004). Thus, an important practical and theoretical challenge is scaling up successful mathematics interventions in the early and primary grades in the United States and especially ensuring the persistence of the effects of such interventions.
The TRIAD model was created to address this challenge. In this study, we evaluated the persistence of effects of a TRIAD implementation, with and without the follow-through component, in its third and final year. That is, two experimental groups received the TRIAD implementation in pre-kindergarten (pre-K), and one of these groups also received the follow-through intervention in their kindergarten and first-grade year. Here we report the effects at the end of the students’ first-grade year and whether these effects were equivalent for various subpopulations, as well as possible indirect, or mediational, effects through particular pedagogical practices (Clements, Sarama, Spitler, Lange, & Wolfe, 2011).
Theoretical Framework
TRIAD’s theoretical framework (Sarama et al., 2008) is an elaboration of the network of influences theory (Sarama, Clements, & Henry, 1998). Scale-up is considered to involve multiple coordinated efforts to maintain the integrity of the vision and practices of an innovation through increasingly numerous and complex socially mediated filters, through phases of introduction, initial adoption, implementation, and institutionalization.
The TRIAD model scales up the support of “interactions among teachers and children around educational material” (Ball & Cohen, 1999, p. 3). This strategy creates extensive opportunities for teachers to focus on mathematics, goals, and students’ thinking and learning, which improves teachers’ knowledge of subject matter, teaching, and learning and increases student achievement (Ball & Cohen, 1999; Cohen, 1996; Schoen, Cebulla, Finn, & Fi, 2003; Sowder, 2007). Research suggests that the most important feature of a high-quality educational environment is a knowledgeable and responsive adult and that professional development can foster these characteristics (Darling-Hammond, 1997; National Research Council, 2001; Sarama & DiBiase, 2004; Schoen et al., 2003; Sowder, 2007). Use of demonstrations, practice, and feedback, especially from coaches, increases the positive effects of information-only training (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005; Pellegrino, 2007; Showers, Joyce, & Bennett, 1987). The professional development in TRIAD provides a promising path for developing teachers’ understanding of learning, teaching, curriculum, and assessment by focusing on research-based models of students’ thinking and learning (cf. Bredekamp, 2004; Carpenter & Franke, 2004; Hiebert, 1999; Klingner, Ahwee, Pilonieta, & Menendez, 2003). Research-based learning trajectories are TRIAD’s core (Clements & Sarama, 2004a, 2004b; Clements, Sarama, & DiBiase, 2003). Learning trajectories have three components: a goal (that is, an aspect of a mathematical domain students should learn), a developmental progression of levels of thinking, and instruction that helps them move along that trajectory. Thus, they facilitate teachers’ learning about mathematics, how students think about and learn this mathematics, and how such learning is supported by the curriculum and its teaching strategies. They address domain-specific components of learning and teaching that have the strongest impact on cognitive outcomes (Lawless & Pellegrino, 2007). By illuminating potential developmental progressions, they bring coherence and consistency to goals, curricula, and assessments (Clements & Sarama, 2009; Sarama & Clements, 2009) and help teachers focus on the “conceptual storyline” of the curriculum, a critical element that is often missed (Heck, Weiss, Boyd, & Howard, 2002; Weiss, 2002).
The full TRIAD model (see Sarama et al., 2008) includes guidelines for promoting equity through the use of curriculum and instructional strategies that have demonstrated success with underrepresented populations (Kaser, Bourexis, Loucks-Horsley, & Raizen, 1999) and promoting communication among key groups around a shared vision (Fixsen, Blase, Naoom, & Wallace, 2009; Fullan, 1992; Hall & Hord, 2001; Huberman, 1992; Kaser et al., 1999; Snipes, Doolittle, & Herlihy, 2002). The model also requires maintaining frequent, repeated assessment (“checking up”) efforts and follow-through efforts emphasizing the purpose and expectations of the project. That is, key groups are involved in continual improvement through cycles of data collection and problem solving (Fixsen et al., 2009; Fullan, 1992; Hall & Hord, 2001; Huberman, 1992; Kaser et al., 1999; Snipes et al., 2002).
The complete TRIAD model also addresses the issue of persistence of effects. Evidence regarding the persistence of early interventions is mixed, with some researchers reporting positive and long-lasting effects (Broberg, Wessels, Lamb, & Hwang, 1997; Garces, Currie, & Thomas, 2002; Gray, Ramsey, & Klaus, 1983; Magnuson & Waldfogel, 2005; Montie, Xiang, & Schweinhart, 2006) and others claiming that the effects fade in the primary grades (Currie & Thomas, 1995; Fish, 2003; Natriello et al., 1990; Preschool Curriculum Evaluation Research Consortium, 2008; Turner, Ritter, Robertson, & Featherston, 2006; U.S. Department of Health and Human Services—Administration for Children and Families, 2010). A recent meta-analysis on fadeout involving nearly 1,100 effect sizes taken from 65 studies reported that impacts are reduced by about .04 standard deviation units per year, which implies that program impacts persist for about 10 years (Leak et al., 2012). Some researchers explicitly or implicitly attribute such fadeout to the evanescence of the effects per se, perhaps due to their inadequate potency (Fish, 2003; Natriello et al., 1990; Turner et al., 2006; U.S. Deptartment of Health and Human Services—Administration for Children and Families, 2010). The TRIAD model is based on the hypothesis that such claims are conceptually flawed and inaccurate and that many present educational contexts (e.g., minimal demands of curricula, standards, and teaching practices) unintentionally undermine persistence of measured effects of early interventions (cf. Brooks-Gunn, 2003). For example, after a successful pre-K experience, children may experience kindergarten and first-grade classrooms in which both the teachers and curricula assume little mathematical competence and target only early-developing skills (even without such pre-K experience, kindergarten and first-grade instruction often covers material children already know; Carpenter & Moser, 1984; Engel, Claessens, & Finch, in press; Van den Heuvel-Panhuizen, 1996). Teachers may remain unaware that some of their students have already mastered the material they are about to “teach” them (Bennett, Desforges, Cockburn, & Wilkinson, 1984; Clements & Sarama, 2009; National Research Council, 2009; Sarama & Clements, 2009; Thomas, 1982). Even if teachers are so aware, pressure to increase the number of students passing minimal competency assessments may lead some teachers to work mainly with the lowest performing students. Within this context and without continual, progressive support, early gains appear to fade. To ameliorate these potentially deleterious factors, TRIAD provided the kindergarten and first-grade teachers in one treatment group knowledge of the pre-K intervention and ways to build upon that knowledge using learning trajectories. This was not a full curricular and pedagogical intervention as we implemented in the pre-K TRIAD classrooms, but rather a test of our hypothesis that helping primary grade teachers build upon the pre-K work would support the persistence of effects. The next section briefly describes the previously reported evaluations of our implementation of TRIAD in students’ pre-K and kindergarten years.
Evaluations of the First 2 Years
The TRIAD implementation occurred in 42 schools in two urban districts serving low-income communities, randomly assigned to three conditions. Here we review the results from students’ pre-K year. We then discuss the issue of the persistence of effects and the rationale for the follow-through components and end with a review of the results from the kindergarten year.
Pre-K year
The largest-scale implementation of the TRIAD model to date was found to be successful at the pre-K level on near and far transfer measures. Students in the TRIAD groups learned more mathematics than the control students (effect size, g = .72; see Clements et al., 2011). Of possible moderators (including school percentage of free/reduced lunch and limited English proficiency, gender, and ethnicity variables), only one was significant: African American students learned less than other students in control classrooms and more than other students in TRIAD classrooms. Three components of a measure of the quantity and quality of classroom mathematics environments and teaching partially mediated the treatment effect: total number of computers on and working for students, the classroom culture component, and the total number of math activities (Clements et al., 2011). The pre-K TRIAD implementation also showed far transfer effects on measures of language competence in the beginning of their kindergarten year (Sarama, Lange, Clements, & Wolfe, 2012).
Kindergarten year
As an initial test of the alternative hypotheses regarding persistence of effects, we evaluated the effectiveness of TRIAD’s follow-through intervention. Kindergarten teachers were taught about what children learned previously and ways to build upon it. They used the districts’ written curricula, but were taught learning trajectories, especially the developmental progressions and how to modify their extant curricula to more closely match the levels of thinking of their students. They also received access to the Building Blocks software (Clements & Sarama, 2007/2012), the same suite that students had used previously, which follows the learning trajectories through the primary grades. We first performed “intent-to-treat” (ITT) analyses, in which the condition group (school) to which the student was originally assigned was maintained as that student’s condition, regardless of how many days the student experienced in that school. These analyses estimate the lower bound of potential effects of the intervention as the focus is on the impact of the random assignment, ignoring anomalies such as crossover of students between treatment conditions. ITT analyses showed that at the end of kindergarten, students in the follow-through condition, but not the non–follow-through condition, scored statistically significantly higher than students in the control condition (g = .33; for a full report, see Sarama, Clements, et al., 2012). We also performed “treatment-on-the-treated” (TOT) analyses that estimate the effects for students who experienced the full duration of the condition to which they were originally assigned. Both groups outperformed the control condition in TOT analyses (g = .38 for the follow-through, g = .30 for the non–follow-through). In these analyses, only one moderator was statistically significant, with African American students within the TRIAD follow-through group scoring significantly better on kindergarten outcomes than African American students in the TRIAD non–follow-through group. The intervention’s effects at the end of kindergarten were mediated by the number of specific mathematical activities and the classrooms’ mathematics culture.
Present Study—Analyses of Effects in First Grade
Given the longitudinal nature of the issues of fading effects, and especially given the substantial difference between mathematics standards and curricula in pre-K and kindergarten versus first grade (e.g., increased formality of the mathematical expectations in first grade) and the possibility that mathematical competencies developed in the pre-K intervention play an even more important role in first grade than in kindergarten (e.g., pre-K learning of numerical and geometric composition and decomposition may particularly support the first-grade focus on addition and subtraction), it is important to evaluate the effects of the TRIAD follow-through intervention into children’s first-grade year. The current analysis examines these effects, utilizing ITT and TOT analyses to investigate differences in mathematics achievement between follow-through, non–follow-through, and control conditions. Given our previous findings, we continue the examination of African American identification as a moderator for the impact of treatment group. Further, although multiple studies have investigated gender as moderator on mathematical competency, the overall results are mixed. Some have found slight advantages for male students within distinct mathematical skills (Royer, Tronsky, Jackson, & Horace Marchant, 1999), in use of abstract strategy use (Fennema, Carpenter, Franke, & Levi, 1998), or at higher grades (Martens, Hurks, Meijs, Wassenberg, & Jolles, 2011). As this analysis considers the sample at first grade and signals a change in the expectations and content associated with mathematics instruction in traditional elementary schools, the current analysis sought to capture any developing differences in mathematical competencies across gender.
At the school level, we include the percentage of children receiving free or reduced lunch. As we purposely sampled within low-resources communities, the majority of our schools had more than 80% of their children receiving assistance for lunch. Also, although individual data on language competency with English was not consistently collected across all schools, we include the school-level percentage of children identified with English as a second language or with limited English proficiency. Further, although 4.6% of children took the pre-K pretest in Spanish and 3.2% took the pre-K posttest in Spanish, none of the children continued to need a Spanish version of the mathematics assessment across the subsequent time points. Finally, we continued to evaluate the mediational effects of the number of specific mathematical activities and the classrooms’ mathematics culture, both statistically significant in prekindergarten and kindergarten, as well as other possible impacts of early childhood classroom environments.
We addressed three primary research questions.
Research Question 1: What is the persistence of effects of the TRIAD intervention, with and without follow-through, on achievement at the end of first grade? Do students in the TRIAD follow-through (TRIAD-FT) group on the average outperform students in the TRIAD non–follow-through (TRIAD-NFT) group in mathematics achievement at the end of first grade? Do students in each of these experimental groups outperform those in the control group?
Research Question 2: Are there significant moderators of any statistically significant effects? Do the effects of the three conditions differ by gender or ethnic group?
Research Question 3: Do measures of the classroom environment mediate effects of different treatments on achievement? Do specific measures of the quantity and quality of the schools’ mathematics environment and teaching mediate the effects of treatment group on mathematics achievement?
Method
We used a cluster randomized trial (at the school level) experimental design to test TRIAD’s impact across the varied settings. We made a list of eligible schools (all those that had not participated in prior Building Blocks or TRIAD research or development projects) and recorded their earliest school-wide standardized mathematics scores (statewide assessments of fourth graders) as an indicator of possible differences in schools’ emphasis on and achievement in mathematics. Schools were rank ordered on these scores separately within each site and blocks created to contain three schools with similar scores. We (publicly, with five observers, including school administrators and project staff members) then assigned each eligible school to one of three treatment groups, selected randomly from the blocks, using a table of random numbers. We used hierarchical linear modeling (HLM), capitalizing on the nested nature of the data, to evaluate the effects of the interventions on students’ mathematics performance trajectories, account for possible variations of the effects among subgroups, and assess the mediational role of the classrooms’ environment and teaching (Raudenbush, 2007, 2008).
Participants and Contexts
Participants were the 1,305 students in 106 classrooms from the original randomly assigned 42 schools in two urban school districts in the Northeast United States (Clements et al., 2011) and the pre-K to first-grade teachers in those schools. These public pre-K classrooms were housed in the same school buildings as the kindergartens and first grades. Table 1 describes these diverse populations. By the end of first grade, 1,127 students from 347 classrooms in 172 schools were tested. Of these, the 1,079 who completed first grade and both components of the assessment were included in the ITT analysis. All 42 schools were represented in the ITT group of 1,079, with the three treatment groups maintaining their original percentages. We used this population for ITT analyses in which the condition group (school) to which the student was originally assigned was maintained as that student’s condition, regardless of how many days the student experienced in that school. Of these students, 750 remained within their randomized school (and thus in treatment condition) from preschool through first grade. We used this subpopulation for TOT analyses. These 750 students also represented all 42 original schools, again with the three research groups maintaining their original percentages.
Demographics of Participating Schools
Note. AA = African American; percentage FRL = percentage of children receiving free or reduced lunch at the school level; percentage LEP = children identified as either an English language learner or as having limited English proficiency; TRIAD-FT = TRIAD follow-through; TRIAD-NFT = TRIAD non–follow-through.
Both districts used a revision of the Investigations (Investigations in Number, Data, and Space, 2008) mathematics curriculum. Both districts had policies and procedures that limited teachers’ flexibility in implementing the Investigations curriculum. For example, both district mathematics departments wrote and disseminated “pacing guides” that dictated what should be taught each week.
The TRIAD Follow-Through Intervention—First-Grade Year
TRIAD staff provided professional development to 40 TRIAD-FT first-grade teachers from the 12 originally randomized follow-through schools for approximately 7 half-day sessions (a total of 32 hours, about 8 of which dealt with data collection and other logistics for the research). Teacher attendance at these sessions averaged 74% (mode, 83%; median, 80%). At the beginning of the academic year, first-grade teachers were informed that some of the students enrolled in their classrooms had experienced the Building Blocks prekindergarten curriculum and were now participating in a longitudinal study. These teachers were expected to teach their district-assigned curriculum, differentiating their teaching as needed to facilitate continued growth in mathematical learning.
We designed the professional development based on research, meeting about once per month, providing curriculum materials, and attempting to engender a sense of “personal satisfaction” (Sarama & Clements, 2009, pp. 348–349; Sarama & DiBiase, 2004). Pedagogical strategies included lecture, demonstration/modeling, observing children’s mathematical thinking and learning, and classroom practice via video, games, and role play (cf. Wolfe, 1991). Professional development began with the mathematical content and developmental progressions for each major mathematical topic, including counting, adding, subtracting, subitizing, shape, length, and early algebra. Following whole group presentations, small groups of teachers focused on instructional tasks from their Investigations units. They presented case studies from their own classrooms, connecting evidence of children’s thinking to the learning trajectories and determining instructional tasks that would enable students to learn the ideas and skills needed to achieve curricular goals. Staff mentors encouraged teachers to use formative assessment based on the learning trajectories. Through monthly visits, the mentors provided classroom-based support for teachers in making individual and classroom-wide software assignments and in differentiating their teaching. The technological component of the intervention featured accessibility to Building Blocks software activities for students, and, for teachers, an online Building Blocks Learning Trajectories (BBLT) Web application. BBLT provides scalable access to the learning trajectories by means of descriptions, videos, and commentaries (Sarama, Clements, et al., 2012; see also TRIADScaleUp.org) and allows teachers to view the learning trajectories through a curriculum or developmental perspective.
In contrast, teachers in the other two conditions, the “business-as-usual” control and the TRIAD-NFT, as well as the teachers in other schools to which students transferred after pre-K, did not receive follow-through activities for first grade.
Measures
School mathematics environment and teaching
The Classroom Observation of Early Mathematics Environment and Teaching (COEMET) was created based on a body of research on the characteristics and teaching strategies of effective teachers of early childhood mathematics and has been employed in previous research (Clements & Sarama, 2008b; Clements et al., 2011; for a review, see Kilday & Kinzie, 2009; Sarama et al., 2008). There are 28 items, all of which are 5-point Likert scales. Assessors blind to treatment group observed and evaluated the mathematics activities for approximately half the day. Subscales include classroom culture, specific mathematics activities (SMA), the overall number of SMAs observed, and the total time during the observation devoted to mathematics instruction. Assessors complete the classroom culture section once to reflect their entire observation. This section includes two subsections, the first of which includes the environment (showing signs of mathematics) and interactions (e.g., adults interacted with and were responsive to children). An example item in the second subsection, “personal attributes of the teacher,” is “The teacher appeared to be knowledgeable and confident about mathematics (i.e., demonstrated accurate knowledge of mathematical ideas and procedures, demonstrated knowledge of connections between, or sequences of, mathematical ideas).” (See Appendix A in the online journal for the Classroom Culture subscale.) Assessors complete an SMA form for each observed mathematics activity, defined as one conducted intentionally by the teacher involving several interactions with one or more students or set up or conducted intentionally to develop mathematics knowledge. An example item in this section is, “The teacher began by engaging and focusing children’s mathematical thinking (i.e., directed children’s attention to, or invited them to consider, a mathematical question, problem, or idea).” The number of SMAs and total time on mathematics are recorded once for each observation.
To protect against drift, project staff led experienced COEMET observers in reviewing the instrument and manual and practicing administering the COEMET on videotaped classroom vignettes, checking agreement on each item. Interrater reliability for the COEMET, computed via simultaneous classroom visits by pairs of observers (10% of all observations, with pair memberships rotated) was 88%; 99% of the disagreements were the same polarity (i.e., if one was agree, the other was strongly agree). Coefficient alpha (interitem correlations) ranged from .95 to .97 for the 28-item instrument, with .74 for the Classroom Culture scale (considered acceptable, especially given the small number of items across diverse indicators; Cicchetti, 1994) and .90 for the SMA scale (Clements et al., 2011; Clements & Sarama, 2008a). Predictive validity was supported by a regression in which the Classroom Observation total score accounted for a significant amount of the variance in students’ posttest achievement scores after accounting for pretest scores, F(2, 1304) = 384.5, p < .001, r 2Δ = .045 (Clements & Sarama, 2008a).
Students’ mathematical knowledge
The Research-based Early Maths Assessment (REMA) (Clements, Sarama, & Liu, 2008) measures core mathematical abilities of students from age 3 to 8 years using an individual interview format, with standardized administration protocol, videotaping, coding, and scoring procedures. Abilities are assessed according to theoretically and empirically based developmental progressions (National Research Council, 2007; Sarama & Clements, 2009). Topics in number include verbal counting, object counting, subitizing, number comparison, number sequencing, connection of numerals to quantities, number composition and decomposition, adding and subtracting, and place value. Geometry topics include shape recognition, shape composition and decomposition, congruence, construction of shapes, and spatial imagery, as well as geometric measurement, patterning, and reasoning. The developmental progression of items within each trajectory as well as the fit of individual items has been reported in earlier research (Clements et al., 2008). The REMA measures mathematical competence as a latent trait in item response theory (IRT), yielding a score that locates students on a common ability scale with a consistent, justifiable metric (allowing accurate comparisons, even across ages and meaningful comparison of change scores, even when initial scores differ; B. D. Wright & Stone, 1979). The 225 items are ordered by Rasch item difficulty; students stop after four consecutive errors on each of the number and geometry sections. Based on the expected growth in mathematical competency from pre-K to first grade, administration at this time point began with item 30 of the number section and item 6 in the geometry section. A basal rule of six in a row correct was employed for both sections.
Training sessions on the REMA included orientation, demonstration, and practice, with a focus on standardized delivery. Subsequent individual practice sessions were taped and critiqued, with 98% to 100% error-free delivery required for certification. All assessment sessions were videotaped and each item coded by a trained coder for correctness and for solution strategy; 10% of the assessments were double-coded. Both assessors and coders were blind to the group membership of the children. Continuous coder calibration by an expert coder (one tape per coder per week) militated against drift. Calibration feedback was sent to coders, alerting them to any variance from coding protocols. Previous analysis of the assessment data showed that its reliability ranged from .93 to .94 on the total test scores (Clements et al., 2008 ); on the present population, the reliability was .92. In addition, the REMA had a correlation of .86 with a different measure of preschool mathematics achievement (Clements et al., 2008), the Child Math Assessment: Preschool Battery (Klein, Starkey, & Wakeley, 2000), and a correlation of .74 with the Woodcock-Johnson Applied Problems subscale for a pre-K–specific subset of 19 items (Weiland et al., 2012).
Procedures
This evaluation is of the final year of the project’s TRIAD implementation (previous procedures and analyses reported in Clements et al., 2011; Sarama, Clements, et al., 2012; Sarama, Lange, et al., 2012). The same staff conducted the teacher training, classroom observations, and student assessments and thus had substantial experience. Further, all assessors were recertified on their respective instruments.
Mainly due to children transferring to other schools, the number of teachers increased from 106 in pre-K to 275 teachers in kindergarten and 347 in first grade; schools expanded from the original 42 to 140 in the kindergarten year and 172 in the first-grade year. To meet financial constraints, we conducted COEMET observations in 93 of these classrooms, overselecting classrooms with larger proportions of children, sampling randomly within blocks where not all classrooms were selected (see details in Sarama, Clements, et al., 2012). We blocked the classrooms by the number of study students in the classrooms and sampled disproportionately, dividing each of four blocks (>8, 5–8, 2–4, or <2 students) into the three conditions: TRIAD-FT (41% of sample), TRIAD-NFT (31% of sample), and control (28% of sample). Original percentages of sample per research group were TRIAD-FT 36%, TRIAD-NFT 34%, and control 30%. Assessors administered the REMA to students in the spring.
Results
Research questions will be addressed in several sections. First, we examine the effects of the TRIAD interventions, compared to the control condition, across pre-K, kindergarten, and first grade. We present both ITT and TOT analyses. Demographic composition of each group is displayed in Table 1. Second, we measure just the effects of the TRIAD-FT treatment by comparing the TRIAD-FT and TRIAD-NFT interventions (covarying out pre-K posttest scores). In both these sections, we assess potential moderators. An indicator variable for block membership is also included in each analysis to control for the slight variation in the fourth-grade math scores of the included schools at the beginning of the study (Kirk, 1982). HLM analyses (Raudenbush, Bryk, Cheong, & Congdon, 2006) included the use of pre-K pretest REMA scores (see Clements et al., 2011) as covariates at both the child level and (mean aggregated) at the school level for the ITT and TOT models. (See Appendix B in the online journal for the HLM model.) The pre-K covariate at the child level was group centered and all other predictors were grand mean centered (Enders & Tofighi, 2007). Effect sizes are given in Hedges g, which accounts for treatment groups of different sizes. It is calculated by dividing the individual predictor beta coefficient resulting from the HLM analyses by the pooled standard deviation of each outcome variable (Hedges & Hedberg, 2007). Cross-level interactions were examined utilizing online interaction utilities (Preacher, Curran, & Bauer, 2006).
Effects of TRIAD Over 3 Years
Intent-to-treat analyses: Main effects
ITT analyses assessed the effect of the TRIAD interventions across three school years, using the pre-K pretest mathematics achievement score as a covariate and the end of first grade mathematics achievement scores as the dependent variable. The unconditional HLM, in which no predictors are included, indicated that about 20% of the variance in first-grade scores lay between schools (σ2 = .386, τ = .094, ρ = .197). The addition of the group centered pretest at the child level and grand centered school aggregated pretest at the school level accounted for 30% more variance in first-grade outcome scores (σ2 = .291, τ = .045, ρ = .134). To control for variability in school demographics, the percentage of children receiving free/reduced lunch and percentage of children identified as having limited English proficiency (LEP) was included. A blocking variable identifying which randomized block each school belonged to also was included. Together, these control variables, representing the baseline model utilized in subsequent analyses, accounted for 6% of the total residual variance in child outcomes (σ2 = .291, τ = .024, ρ = .075).
In the process of refining our final model, multiple child-level moderators were tested. Of the multiple ethnic groups represented within our sample, children identifying as African American constitute over half of the overall ITT sample (54%). Further, the comparison of African American to an amalgamated “other” including Caucasian (19%), Hispanic (21%), Asian (4%), and Other (2%) was the only ethnic comparison to demonstrate a significant group difference across models. Gender was also conserved for analysis across models as an exploratory factor. The final highly parsimonious model therefore included a dichotomized variable for female or not and another variable for African American or not included at the child level. Further, interactions with the treatment groups were included in each model estimated as predictors at the school level on the child-level slopes. Treatment groups (e.g., TRIAD-FT, TRIAD-NFT, and control) were entered at the school level as dummy-coded indicators. As this research focuses on the comparison of three groups, a system with a single dummy code would not allow the examination of the between-treatment effects in conjunction with each treatment group as compared to the control group. Therefore, two parallel analyses were conducted on each of the following ITT and TOT models. In the left hand panel (LP) of Table 2, to capture the comparison of the TRIAD-FT and control conditions relative to the TRIAD-NFT condition, two dummy-coded variables were utilized wherein in the first treatment variable, TRIAD-FT, was coded as 1 and control and TRIAD-NFT as 0 and in the second treatment variable, control was coded as 1 and TRIAD-FT and TRIAD-NFT as 0. TRIAD-NFT was chosen as the comparative conditions in this panel for ease of interpretation regarding our hypothesis of higher first-grade outcome scores for the TRIAD-FT condition and lower scores for the control condition. In the right hand panel (RP) of Table 2, the dummy coding system was modified to compare the control condition to both the TRIAD-FT and TRIAD-NFT conditions. In this model, the dummy code for the first treatment variable was coded as 1 for TRIAD-FT and as 0 for control and TRIAD-NFT. Similarly, for the second treatment variable, the TRIAD-NFT condition is coded as 1 and the control and TRIAD-FT conditions were coded as 0. This allowed us to examine our additional hypothesis that both treatment groups would demonstrate significantly higher outcome scores at the end of first grade than the control condition. Model fit tests for allowing random effects for slopes were not significant across test models; therefore, fixed effects from each subsequent model are discussed and displayed.
Intent-to-Treat (ITT; N = 1,079) Hierarchical Linear Modeling Final Fixed Effects Model Outcomes and Variance Components for Research-based Elementary Math Assessment (REMA) Mathematics First-Grade Posttest Scores, With Pre-K Scores as Covariates at Both Levels
Note. SE = standard error; SD = standard deviation; coefficient = unstandardized beta; TRIAD-FT = Building Blocks in pre-K with follow-through in kindergarten and first grade; TRIAD-NFT = Building Blocks in pre-K only; AA = African American; LEP = percentage of children identified as English language learner or with limited English proficiency; F/RL = percentage of children receiving free/reduced lunch; block = randomization block identifier based on fourth-grade math scores.
p < .05. **p < .01.
In the final ITT model (Table 2), both the pretest and pretest aggregate were significant predictors of child outcomes at the end of first grade. A small but significant impact for the school-level percentage of children receiving services as an English language learner was found (β = .005; SE = .001, p = .001). As this factor was not included in the original randomization and differences exist across conditions, its significance is not surprising. In addition, as this variable represents school-level percentage of children receiving services, the process by which this relates to overall child outcomes would require additional investigation. Still, in its current role as a control variable, its impact (β = .005) on children outcome scores is quite small relative to other covariates (i.e., pretest scores β = .634). Neither the percentage of children receiving free/reduced lunch nor the blocking identifier accounted for significant variance in mathematics scores.
A significant difference was found for each of the treatment group comparisons. Children within the TRIAD-FT condition significantly outperformed both the TRIAD-NFT (LP: β = .124; SE = .056, p = .03, g = .18) and control (RP: β = .264; SE = .057, p < .001, g = .38) conditions within the ITT sample on the mathematics outcome at the end of first grade. The TRIAD-NFT condition also demonstrated significantly higher mathematics scores relative to the control condition (RP: β = .139; SE = .054, p = .01, g = .21). Overall, the positive impact of the TRIAD-FT condition relative to the control condition is nearly two times larger than the statistically significant but smaller impact of TRIAD-NFT condition relative to control. Thus, the TRIAD-FT condition, as in previous research, continued to demonstrate a significantly larger impact on children outcome scores than the TRIAD-NFT condition.
Gender was not found to be a significant moderator within any of these comparisons. That is, within the ITT sample, treatment was found to be equally effective for children identifying with either reported gender. A main effect for African American was found to be significant (LP/RP: β = –.252; SE = .040, p < .001, g = .39). Overall, children identifying as African American performed significantly worse on the mathematics outcome as compared to children identifying with other ethnic groups. A significant difference, however, for children identifying as African American within the TRIAD-FT condition relative to control was found (RP: β = .191; SE = .094, p = .043). Children identifying as African American within the TRIAD-FT condition scored significantly higher than children identifying as African American within the control condition (relative to children identifying with another ethnic group), yielding a moderate effect size (g = .31). Overall, random assignment to the TRIAD-FT condition significantly reduces the negative main effect for African Americans found across conditions.
Treatment-on-the-treated analyses
All other analyses were conducted on the TOT sample; that is, they were performed on those students who stayed within their randomized condition and school, completed all components of the assessments, and did not repeat or skip kindergarten (N = 750). Table 3 presents the means and standard deviations of mathematics (REMA) scores by treatment condition and time point. An unconditional model based on the TOT sample indicated that 27% of the variance lay across schools (σ2 = .346, τ = .126, ρ = .267). An additional 25% of the variance was accounted for by the addition of the group centered pretest at the child level and school aggregated grand centered pretest at the school level (σ2 = .273, τ = .079, ρ = .225). Finally, the addition of the control variables (e.g., blocking identifier, school level percentage of free/reduced lunch, and school level percentage of English language learners) accounted for an additional 9% of the total residual variance (σ2 = .273, τ = .049, ρ = .154).
Means and Standard Deviations of Mathematics Outcome Scores by Treatment Condition and Time Point
Note. Rasch scores converted to a T-score (M = 50, SD = 10). TRIAD-FT = TRIAD follow-through; TRIAD-NFT = TRIAD non–follow-through.
The same child-level predictors of gender (i.e., female or not) as well as African American (i.e., African American or not) were examined within the TOT sample at the child level. This decision was based upon these two comparisons representing the largest subgroups within the sample. Within the TOT sample, females constituted 53% and children identifying as African American constituted 57% of the first graders remaining within their randomized school. Lastly, treatment indicators were again entered as dummy-coded variables representing the three treatment groups. The same comparison of dummy-coded treatment variables previously described for Table 2 is depicted in Table 4. In the left hand panel, the TRIAD-FT and control conditions are compared to the TRIAD-NFT condition. In the right hand panel the TRIAD-FT and TRIAD-NFT conditions are compared to the control condition. Final TOT model parameter estimates for both two-group comparisons are displayed in Table 4.
Treatment on the Treated (TOT; N = 750) Final Fixed Effects Model Outcomes and Variance Components for Research-based Elementary Math Assessment (REMA) Mathematics First Grade Posttest Scores, With Pre-K Scores as Covariates at Both Levels Including Each Treatment Group Comparison
Note. SE = standard error; SD = standard deviation; coefficient = unstandardized beta; TRIAD-FT = Building Blocks in pre-K with follow-through in kindergarten and first grade; TRIAD-NFT = Building Blocks in pre-K only; block = randomization block identifier based on fourth-grade math scores; LEP = percentage of children identified as English language learner or with limited English proficiency; F/RL=percentage of children receiving free/reduced lunch; AA = African American.
p < .05. **p < .01.
Again, both the child-level and school-level pretest aggregate scores were significant predictors at the end of first grade. Of the other control variables, the only significant main effect was found for the percentage of children receiving services for English language learning (β = .006; SE = .002, p = .004).
As in the ITT analyses, each of the treatment groups was significantly different from one another within the TOT sample. Relative to both the TRIAD-NFT (LP: β = .163; SE = .081, p = .046, g = .24) and control (RP: β = .346; SE = .080, p < .001, g = .51) conditions, children within the TRIAD-FT treatment group demonstrated the greatest mathematics competency at the end of first grade. Children within the TRIAD-NFT condition also demonstrated higher scores than the control group at the end of first grade (RP: β = .183; SE = .080, p = .028, g = .28). Within this sample, the impact of the TRIAD-FT condition relative to control was nearly twice that of the TRIAD-NFT condition relative to control. Attenuating the impact of the TRIAD-FT condition relative to the TRIAD-NFT condition by focusing on only those children who received the full treatment yields a one-third increase in effect as compared to the ITT estimates.
Gender did not act as a significant moderator in either set of comparisons, again suggesting that both boys and girls benefit similarly from exposure to the TRIAD-FT intervention. A significant moderate main effect for African American compared to other ethnic groups was found across both comparison sets (LP/RP: β = –.240; SE = .049 p < .001, g = .38). Across research groups, students identifying as African American scored lower on the assessment of mathematics achievement at the end of first grade controlling for pretest scores as compared to students identifying with other racial/ethnic groups. A significant interaction with the TRIAD-FT and control group comparison, however, was found for students identifying as African American compared to students identifying with other ethnic backgrounds (RP: β = .290; SE = .120, p = .017, g = .38). This moderate effect size indicates that children identifying as African American differentially benefited from follow-through relative to children identifying as African American within the control condition. Focusing on only those students who have received the full treatment, inclusion within the TRIAD-FT condition eliminates the negative overall main effect on first-grade outcome scores for children identifying as African American. The results for these comparisons are displayed in Table 4.
Effects of (Only) the TRIAD Follow-Through Component
As the TRIAD-FT and TRIAD-NFT groups received the same intervention during the pre-K year, we calculated a third two-level model that included the pre-K posttest as a covariate at both the child and school levels to determine the isolated effect of the additional training received by the kindergarten and first-grade teachers within the TRIAD-FT condition, compared to the pre-K only experience of the TRIAD-NFT group (see Table 5). Beginning with the same model including pretest covariates only (σ2 = .273, τ = .079, ρ = .225), the addition of the end of pre-K posttest at both levels of the analysis accounted for a substantial amount of variance, an additional 57% of the variance in child outcome scores (σ2 = .182, τ = .046, ρ = .200). Finally, the same control variables representing the percentage of free/reduced lunch (F/RL), percentage of children receiving services for English language learning, and the blocking identifier were entered at the school level (σ2 = .183, τ = .032, ρ = .151), accounting for an additional 6% in the total residual variance. Thus, the posttest at the end of pre-K is a stronger predictor of end of first-grade math scores than the child’s pre-intervention exposure level of mathematics. Further, this time point reflects one full year of treatment exposure for both the TRIAD-FT and TRIAD-NFT groups (i.e., the pre-K year). The final model included the same child-level moderators (i.e., gender and African American) as well as the dummy-coded treatment indicators. In this model, the TRIAD-FT and TRIAD-NFT conditions are compared to one another. To maintain inclusion of all schools and conditions, the TRIAD-FT variable was dummy coded as 1 and TRIAD-NFT and control as 0. The control variable was coded as 1 for control and 0 for TRIAD-NFT and TRIAD-FT conditions. Included together, the compared condition for this analysis is the TRIAD-NFT condition.
“Value Added” Treatment on the Treated (TOT; N = 750) Final Fixed Effects Model Outcomes and Variance Components for Research-based Elementary Math Assessment (REMA) Mathematics First-Grade Posttest Scores, With Pretest and Posttest Pre-K Scores as Covariates at Both Levels Comparing TRIAD-FT to TRIAD-NFT
Note. SE = standard error; SD = standard deviation; coefficient = unstandardized beta; TRIAD-FT = Building Blocks in pre-K with follow-through in kindergarten and first grade; TRIAD-NFT = Building Blocks in pre-K only; block = randomization block identifier based on fourth-grade math scores; LEP = percentage of children identified as English language learner or with limited English proficiency; F/RL = percentage of children receiving free/reduced lunch; AA = African American.
p < .05. **p < .01.
A significant difference between the TRIAD-FT and TRIAD-NFT conditions, controlling for both pretest and posttest, was found (β = .169; SE = .075, p = .032, g = .25). The TRIAD-FT condition demonstrated significantly higher outcome scores at the end of first grade relative to the TRIAD-NFT condition. This moderate effect size is comparable to the impact of the TRIAD-NFT condition relative to the control condition across the ITT and TOT models. Providing continuing professional development focused on mathematics learning trajectories to kindergarten and first-grade teachers helped children maintain the benefits of the pre-K intervention.
Of the control variables, only the percentage of children receiving services as an English language learner demonstrated a significant main effect at the school level (β = .005; SE = .002, p = .037). Although gender failed to demonstrate a significant main effect, the interaction between gender and treatment group, comparing TRIAD-FT to TRIAD-NFT, approached significance (β = .145; SE = .076, p = .056). Females within the TRIAD-FT condition outperformed the males within both the TRIAD-NFT and control conditions (g = .21, Table 5).
A significant main effect was found for African American (β = –.167; SE = .042, p < .001, g = .26), whereby students who identify as African American demonstrated lower mathematics achievement scores at the end of first grade controlling for both pretest and posttest as compared to students identifying as a member of another ethnic/racial grouping. The interaction with the TRIAD-FT condition, however, was not significant (β = .068; SE = .101, p = .504). Therefore, when focusing on growth between kindergarten and first grade, there was no evidence that the TRIAD-FT treatment differentially supported African American students.
Mediation
We hypothesized that aspects of the classroom environment, as measured on the COEMET, serve to mediate the impact of the treatment condition on student outcomes, specifically, that the intervention (i.e., TRIAD-NFT and TRIAD-FT) operates through aspects of the classroom environment on student outcome mathematics scores collected at the end of first grade. Our analysis focused on three aspects of the COEMET that have previously been found to exhibit significant indirect effects on student outcomes on the same mathematics measure at the end of prekindergarten and kindergarten: the number of specific math activities, the quality of the specific math activities, and the classroom culture score. Again, to control for prior experience and improve our treatment effect outcomes, the pretest scores were covaried at both the student and school level in the mediational analysis. For ease of interpretation, to relate to our earlier findings, and to provide an upper limit estimate on possible indirect effects for classroom-level variables, these results present only pretest covariates. Further, the current two-level analysis of student nested within school necessarily obscures individual teacher differences within schools that may account for and allow for the investigation of other process level variables. Additional covariates would reduce the effect for each of the following models and represents a limitation in this analysis. These estimates therefore represent an upper bound on the impact of the measured school-level variables on child outcome scores at the end of first grade.
Utilizing notation (m) and (y) to reference the mediator and the outcome (Pituch, Stapleton, & Kang, 2006), the standard regression equation for the impact of the treatment group on the mediator can be
where M
j
and X
j
represent the school-level mediator and treatment condition and a represents the impact of the treatment condition on the mediator. The intercept and residual for the equation are estimated as
where Y
ij
is the child outcome at first grade posttest,
where
The a and b Path Coefficients for Kindergarten Classroom Observation of Early Mathematics Environment and Teaching (COEMET) Mediational Models, Including Significance Level of Each Path and 95% Confidence Intervals (CI) for the Indirect Effects
Note. Significant indirect effects are marked in bold. TRIAD-FT = TRIAD follow-through; TRIAD-NFT = TRIAD non–follow-through; SMAs = specific math activities.
p < .05.
We hypothesized that the TRIAD-FT condition would demonstrate greater indirect effects through these aspects of the classroom environment as this group of students continued in classrooms in which the teachers were instructed in the intervention. Increasing the quantity and quality of mathematics within the classroom environment is a primary focus of the intervention. Our results confirmed that the classroom components did not demonstrate any significant indirect effects for the TRIAD-NFT condition relative to control. The TRIAD-FT condition, however, did have significant indirect effects. A significant 20% of the total effect of the TRIAD-FT condition, relative to TRIAD-NFT, on student outcome scores was transmitted by scores on the classroom culture from the student’s experience in kindergarten (TRIAD-NFT indirect effect, .148; CI [.021, .328]). Similarly, as compared to the control condition, 35% of the total effect of the TRIAD-FT condition on outcome scores was transmitted through the classroom culture (indirect effect: .118; CI [.011, .274]). Exposure to the intervention during kindergarten served to improve the mathematical culture of the classroom, thereby increasing child outcome scores at the end of first grade. The other two components of the COEMET tested within this model, quality and quantity of specific math activities, did not demonstrate significant indirect effects. Relative to the control condition, however, the TRIAD-FT condition demonstrated significantly more specific math activities, though the indirect effect was not significant. None of the indirect effects examined for these COEMET components measured at first grade were found to be significant. The estimates for the a and b paths comprising each indirect effect for the kindergarten COEMET mediational analyses and their 95% confidence intervals are presented in Table 6.
Discussion
We need generalizable models of scale-up and longitudinal research evaluating the persistence of the effect of their implementations. This study was a cluster randomized trial evaluation of the persistence of effects of a research-based model for scaling up educational interventions, with and without a follow-through intervention, into its third year of implementation. The instantiation of the TRIAD (Technology-enhanced, Research-based Instruction, Assessment, and professional Development) model was designed to teach early mathematics for understanding emphasizing learning trajectories and technological tools. The interventions were identical at pre-K, but only one included TRIAD’s follow-through component in students’ kindergarten and first-grade years.
We classified results into four categories. First, we discuss the effects of the interventions on mathematics achievement at the end of first grade, with intent-to-treat analyses that included all participating students, regardless of how much of the intervention they experienced. Second, we discuss the same effects with treatment-on-the-treated analyses that included only those students who experienced the assigned treatments. Third, we report whether these effects differed for different groups by examining several possible moderators of the effects. Fourth, we examined which, if any, measured components of the quantity and quality of the classroom environments and teaching mediated the effects of the TRIAD interventions.
First, addressing effects on students’ mathematics achievement at the end of their first-grade year, the ITT analyses assessed the effect of the TRIAD interventions over the three school years, including all available students, regardless of whether they had switched classrooms, schools, or conditions. Results showed a significant difference between the TRIAD follow-through and the control group (g = .38) as well as the TRIAD-FT versus TRIAD non–follow-through (g = .18) comparison. The TRIAD-NFT and control conditions (g = .21) were also statistically significantly different. Even under the effects of dilution through mobility, the TRIAD intervention demonstrated significantly higher scores than the business as usual control condition and the follow-through component caused significantly higher scores than the non–follow-through condition.
Second, TOT analyses were conducted on all students who remained in their randomized condition throughout the 3 years. We contrast the ITT and TOT approaches, as the focus of this study is the effects of an intervention defined by consistently receiving mathematics instruction from educators engaged in professional development based upon learning trajectories of development within mathematics. Recall that at the end of pre-K, the TRIAD groups (TRIAD-FT and TRIAD-NFT, identical at pre-K) were statistically significantly higher in mathematics achievement than the control group (Clements et al., 2011). By the end of kindergarten, both groups similarly outperformed the control group. Our results at the end of first grade continue to support the benefits of high-quality, research-based mathematics instruction during pre-K as both groups continue to outperform the control conditions in TOT analyses. That is, with just the pre-K component, the TRIAD-NFT condition demonstrated significantly higher mathematics achievement than the control condition (g = .28). Further, similar research-based instruction throughout the early years is more beneficial. At the end of first grade, the TRIAD-FT condition had significantly higher mathematics achievement than both the control condition (g = .51) and the TRIAD-NFT condition (g = .24).
Thus, extending the TRIAD intervention with follow-through to the end of first grade maintained the statistically significant gains with about the same effect sizes as measured at the end of kindergarten. In addition, these children scored significantly higher than those whose teachers implemented the pre-K but not the follow-through intervention. This evidence supports the persistence of the effects of this extended implementation of the TRIAD model, indicating that the follow-through component is important for maintaining the learning trajectory engendered by the pre-K intervention. Without the follow-through component, the effects are smaller each year.
Third, we examined potential moderators of gender and ethnic group (preliminary analyses confirmed previous results of no significant interaction with school percentage of free/reduced lunch). Multiple analyses revealed interactions between the TRIAD-FT intervention and African American students versus students identified with other ethnic groups. African American students within the TRIAD-FT group scored significantly better on student first-grade outcomes than African American students in the control group. Centering instruction around learning trajectories may focus teachers’ attention on students’ thinking and learning of mathematics rather than their memberships in ethnic groups and thus avoids perceptions that negatively affect teaching and learning (McLoyd, 1998; Pallas, Alexander, Entwisle, & Thompson, 1987; U.S. Deptartment of Health and Human Services — Administration for Children and Families, 2010).
Examining only the effects of the follow-through intervention (children’s kindergarten and first-grade years only, with pre-K posttest scores as a covariate), the TRIAD-FT condition exhibited greater mathematical competency than children within the TRIAD-NFT condition (g = .25). This indicates that the additional training provided for the kindergarten and first-grade teachers of these students continued to support children’s development of mathematical competency. Overall, no interactions between ethnicity and the TRIAD-FT and TRIAD-NFT comparison were significant. That is, although all analyses that include the pre-K year indicate that African Americans score higher the more of the TRIAD intervention they experience, this interaction was not significant for analyses including only the kindergarten and first-grade years. This suggests that to continue to close the achievement gap between students who identify themselves as African American and other ethnic groups, additional work with primary grade teachers may be necessary, a point to which we will return. Finally, the appearance of a trend suggesting greater growth in mathematical competency for girls may warrant attention in future research. That is, given the change in mathematical expectations in first grade, documented differences in strategy use (Fennema et al., 1998), the near significant finding for the follow-through component (p = .056), and the effect size of this interaction (g = .21), further investigation is needed.
Fourth, we examined what components of the classroom observation instrument mediated the effects. Only one statistically significant component of the instrument was found. A moderate portion of the total effect of the TRIAD follow-through intervention on student outcomes, relative to both the TRIAD-NFT and control conditions, was transmitted through the classroom culture observed in kindergarten. This is consistent with findings from the pre-K and kindergarten years. Exposure to the follow-through intervention may have created a greater focus on mathematics in these classrooms, which in turn increased student achievement. Such mediation is consistent with the literature indicating that learning is influenced by features of the classroom, especially overall mathematical activity and teachers who are not only knowledgeable and enthusiastic about mathematics, but also who interact with and respond to students’ mathematical thinking (Clarke & Clarke, 2004; Clements & Sarama, 2007a, 2009; Fraivillig, Murphy, & Fuson, 1999; Sawada et al., 2002; S. P. Wright, Horn, & Sanders, 1997). Further, other components of the classroom observation that were significant mediators in previous work were not significant in this study. These results may be due simply to the limited number of classroom observations we could make across the large number of first-grade classes.
Implications
From the multiple perspectives of theory, research, policy, and practice, there is a need for transferable, empirically supported models of scaling up successful interventions (Borman, 2007; Cuban & Usdan, 2003; McDonald et al., 2006), particularly targeting young children’s learning of mathematics (National Mathematics Advisory Panel, 2008). Multiple studies have supported the effectiveness of the TRIAD model (Clements et al., 2011; Sarama, Clements, et al., 2012; Sarama et al., 2008; Sarama, Lange, et al., 2012). This study provides the most compelling evidence to date regarding the importance of follow-through. With such follow-through, the effects from the pre-K intervention persisted; without follow-through, they were significantly smaller.
This finding has implications for the field. Multiple researchers have reported that preschool benefits do not persist; that is, that gains “fade” (Fish, 2003; Leak et al., 2012; Natriello et al., 1990; Preschool Curriculum Evaluation Research Consortium, 2008; Turner et al., 2006; U.S. Department of Health and Human Services—Administration for Children and Families, 2010)—a main rationale for the follow-through component and this study. We believe that such an interpretation mistakenly treats initial effects of interventions as independent of the students’ future school contexts. That is, these researchers theoretically reify the treatment effect as an entity that should persist unless it is “weak” and thus susceptible to fading. Such a perspective identifies the gain as a static object carried by the student that, if not evanescent, would continue to lift the student’s achievement above the norm. Our theoretical position and this study’s empirical results support an alternative view. Successful interventions do provide students with new concepts, skills, and dispositions that change the trajectory of the students’ educational course. However, these are, by definition, exceptions to the normal course for these students in their schools. Because the new trajectories are exceptions, multiple processes may vitiate their positive effects, such as institutionalization of programs that assume low levels of mathematical knowledge and focus on lower level skills and cultures of low expectations for certain groups (and, as noted, kindergarten and first-grade instruction often covers material children already know even without pre-K experience; Carpenter & Moser, 1984; Engel et al., in press; Van den Heuvel-Panhuizen, 1996). Left without continual, progressive support, children’s nascent learning trajectories revert to their original, limited course. An alternative explanation is that a stronger initial (pre-K) intervention is necessary to counteract any early disadvantage in requisite readiness skills. However, such an argument has been called unrealistic, especially if children attend poor-quality schools (Brooks-Gunn, 2003), which is more likely for African American students (Currie & Thomas, 2000). There is a cumulative positive effect of students experiencing consecutive years of high-quality teaching and a cumulative negative effect of low-quality teaching (Ballou, Sanders, & Wright, 2004; Jordan, Mendro, & Weerasinghe, 1997; Sanders & Horn, 1998; Sanders & Rivers, 1996; S. P. Wright et al., 1997). The latter is more probable for high-risk children (Akiba, LeTendre, & Scribner, 2007; Darling-Hammond, 2006). Further, the maintenance of the effect size in the TRIAD follow-through intervention, compared to the decreasing effect size in the non–follow-through intervention, militates against such an interpretation.
If supported by additional research, the finding has implications for both theory and policy. Interpretations of this fade may call for decreased funding and attention to pre-K (Fish, 2003, 2007). Although this may appear reasonable (e.g., “If effects of an intervention fade out, why fund that intervention?”), this ignores future school contexts. Instead, if such effects “fade” in traditional settings but do not in the context of follow-through interventions, then attention to and funding for interventions in both pre-K and the primary grades should arguably increase. This position is consistent with that of (a) the authors of the meta-analyses on fadeout, who conclude that because it takes a long time (about 10 years) for impacts to disappear, there is more than enough time for possible follow-through interventions that capitalize on the gains from these programs (Leak et al., 2012) and (b) intervention researchers’ notion of environmental maintenance of development (see also Cooper, Allen, Patall, & Dent, 2010; Ramey & Ramey, 1998). The follow-through component in this study cost about $1,900 per teacher (this includes the costs of the professional development staff, substitute teachers, and separately funded coaches for the year). More extensive and effective interventions are needed and may be achievable with similar funding.
In the evaluation of the same students in this study as well as previous studies, the TRIAD implementation was relatively more successful for students who identified themselves as African American than other ethnic groups. Although African American students continued to lag behind non–African American students in all conditions, the TRIAD-FT intervention helped them narrow that achievement gap. A high-quality, consistent mathematics education with an emphasis on learning trajectories can make a demonstrative and consistent positive impact on the educational attainment of African American students in the pre-K, kindergarten, and first-grade years compared to traditional instruction that is not accompanied by professional development based on student learning trajectories. We did not hypothesize this interaction, so we proffer explanations that are by necessity post hoc. (a) Centering instruction around learning trajectories may focus teachers’ attention on students’ thinking and learning of mathematics and what children can learn to do (Celedón-Pattichis, Musanti, & Marshall, 2010), avoiding biases, such as views of African American students’ learning from a deficit perspective (Rist, 1970), that impair teaching and learning (Martin, 2007; McLoyd, 1998; Pallas et al., 1987; U.S. Department of Health and Human Services—Administration for Children, 2010). That is, especially given the significant mediation of the classroom culture, including enthusiastic interaction with children around mathematics they believe children can learn, it may be that the TRIAD interventions changed teachers’ views of African American students’ mathematical capabilities (Jackson, 2011). The curriculum’s learning trajectories are based on the notion that learning is developmental and amenable to instruction, and the curriculum’s approach, including specific, sequenced activities and formative assessment strategies, may have offered a way to act on these nascent views. In such action, the productive views are further strengthened. (b) The TRIAD intervention may promote a conceptual and problem-solving approach infrequently emphasized in schools serving low-income children (Stipek & Ryan, 1997), explicitly supporting African American students’ participation in increasingly sophisticated forms of mathematical communication and argumentation (e.g., asking “How do you know?” as opposed to the “pedagogy of poverty” frequently used with African American students; Jackson & Wilson, 2012; Ladson-Billings, 1997). (c) The TRIAD follow-through intervention may raise several aspects of the quality of mathematics education, lack of which has been suggested as a reason preschool benefits dissipate for African American children (Currie & Thomas, 1995; see also Zhai, Raver, & Jones, 2012); for example, the language-rich nature of the curriculum and its expectation that all children invent solution strategies (cf. Carr, Steiner, Kyser, & Biddlecomb, 2008; Fennema et al., 1998) and explain them (such representations and explanations promote future mathematics learning; Siegler, 1995). These and other possible reasons should be evaluated, compared, and combined, especially in interventions targeted to the primary grades.
The TRIAD follow-through intervention’s effect was partially due to the increase in the positive classroom cultures teachers develop, at least in the kindergarten classrooms. Interventions such as TRIAD may help engender a greater focus on mathematics, which in turn can help increase students’ mathematics achievement. As other work has shown (Carpenter, Fennema, Peterson, & Carey, 1988; Clements et al., 2011; Jacobs, Franke, Carpenter, Levi, & Battey, 2001; National Research Council, 2009), helping primary teachers gain additional knowledge of mathematics, students’ thinking and learning about mathematics, and how instructional tasks can be designed and modified—that is, the three components of learning trajectories—has a measurable, positive effect on their students’ achievement. This is particularly important in the early years because teachers often do not recognize when tasks are too difficult, but even when they do, they provide “more of the same” (Bennett et al., 1984). Further, they overlook tasks that provide no challenge to children—that do not demand enough (Bennett et al., 1984; Van den Heuvel-Panhuizen, 1996). Thus, most children, especially those who have some number knowledge, may learn little or no mathematics in kindergarten (B. Wright, 1991).
Implementing interventions such as TRIAD is therefore important, given that early mastery of concepts and skills in mathematics and literacy is the best predictor of students’ successful academic careers (Aunola, Leskinen, Lerkkanen, & Nurmi, 2004; Duncan, Claessens, & Engel, 2004; Duncan & Magnuson, 2011; Paris, Morrison, & Miller, 2006; Stevenson & Newman, 1986). Further, students from low-income communities benefit more relative to students from higher resource communities from the same “dose” of school instruction (Raudenbush, 2009). Thus, comprehensive implementations of research-based models, such as the TRIAD follow-through model, may be especially effective in such lower resource schools. This speaks to a caveat concerning the effectiveness of the TRIAD follow-through intervention. The intervention maintained, but did not add to, the gains of the more comprehensive TRIAD pre-K intervention. Differences in scores remain statistically significant, but effects were not cumulative. Future design studies might investigate ways to (a) avoid or ameliorate the limiting influence of pacing guides and other school district policies that may have limited the effect of the TRIAD follow-through component, (b) increase the intensity or duration of that component, or (c) implement different and more extensive interventions, such as curriculum replacement (as the TRIAD intervention did in pre-K). That is, future research should evaluate the efficacy and scalability of a fully implemented TRIAD model in the primary grades to see if the pre-K slope can be maintained throughout elementary school. Such an intervention may go beyond “resisting fade out” to show that a positive rate of learning can and should be sustained. In other words, we argue that what should persist is not just a pre-K gain, but also a dramatic trajectory of successful learning.
Findings for this and other implementations of the full TRIAD model, along with limitations of these studies, have five additional research implications. First, our research design could not identify which components of the TRIAD model and its instantiation are core components. Such research would be theoretically and practically useful. Second, in a related vein, the finding that TRIAD was particularly beneficial for African American children needs to be tested and, if replicated, explained. We proffered three possible reasons that might be tested in future research.
Third, this study addressed the persistence of TRIAD’s effects. We also need studies of sustainability, the length of time an innovation continues to be implemented with fidelity (cf. Baker, 2007), especially given the “shallow roots” of many reforms (Cuban & Usdan, 2003). We presently are collecting such data.
Fourth, TRIAD was designed as a general model of scaling up successful interventions. TRIAD’s 10 research-based guidelines are consistent with but more detailed than generalizations from the empirical corpus (Pellegrino, 2007). However, the model has not been implemented outside of the early childhood age range, or outside of urban districts with high percentages of students from low-resource communities, nor in other subject-matter domains. Evaluations of such varied implementations are needed. Fifth, the present study supports a guideline of the TRIAD model, the use of learning trajectories. That is, compared to the pre-K intervention’s inclusion of a new curriculum with multiple components, the kindergarten and first-grade interventions included only one new curriculum component, the Building Blocks software. The core of the follow-through intervention was developing teachers’ knowledge of the mathematical learning trajectories for their grade level. Thus, the TRIAD studies join the growing research corpus that supports the educational usefulness of learning trajectories, including evaluations of curricula built upon learning trajectories (Clements & Sarama, 2007b, 2008a; Sarama et al., 2008), elementary curricula based on related trajectories (e.g., Math Expressions in Agodini & Harris, 2010), studies of successful teaching (Wood & Frid, 2005), and professional development projects (Bright, Bowman, & Vacc, 1997; Clarke et al., 2002; R. J. Wright, Martland, Stafford, & Stanger, 2002). This supports the use of such structures in standards, such as the recently released Common Core State Standards (CCSSO/NGA, 2010). Again, however, the specific contribution of the learning trajectories per se needs to be disentangled and identified.
Footnotes
Notes
D
J
C
M
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
