Protocol for a Systematic Review: “No Excuses” Charter Schools for Increasing Math and Literacy Achievement in Primary and Secondary Education: A Systematic Review and Meta-Analysis

Abstract

Linked article:

BACKGROUND

The Problem: Persistent academic achievement gaps

The prevalence of racial and income-based educational achievement gaps is one of the most critical current issues in U.S. education and contributes significantly to racial and economic inequality (Duncan & Magnuson, 2011). Racial and economic disparities develop in large part because children from low-income or minority (i.e., Black and Hispanic) households disproportionately manifest poor academic achievement in school, leading to lifelong consequences such as lower earned incomes in adulthood relative to peers from higher-income and/or racial majority households. According to the Achievement Gap Initiative at Harvard University, disparities in family resources predict between one-half and two-thirds of the total racial achievement gap (Achievement Gap Initiative, 2013). Poverty rates in 2010 were 27.4% for Black and 26.6% for Hispanics, compared to 9.9% for non-Hispanic whites and 12.1% for Asians (National Poverty Center, 2010). Academic achievement gaps are both the causes and the consequences of the disparity of resources.

According to the National Assessment of Education Progress Long Term Trend Assessment (NAEP; U.S. Department of Education, 2014), the academic performance of both 9- and 13-yeard-olds has improved across all groups since the 1970s, and gaps between groups have narrowed. However, the gaps between Black and Hispanic 17 year-olds and their white peers, which narrowed during the 1970s and 1980s, have remained stable since 1990 without explanation, and the income-based achievement gap has widened significantly over the last three decades (Reardon, 2013). Despite periods of progress, racial and income-based achievement gaps remain substantial (Barton & Coley, 2010; Jacob & Ludwig, 2008). They begin early in life, and, given the rapidly growing population among the lowest-achieving groups (U.S. Census Project, 2010), these gaps will only become more critical in the future.

No Excuses Charter Schools

Charter schools in urban communities across the United States have begun employing a No Excuses philosophy, which focuses intensely on improving the math and literacy achievement of students who come primarily from low-income and minority backgrounds (Cheng, Hitt, Kisida, Mills, 2015). The formal discussion of schools defined as “No Excuses” began as a national effort organized by The Heritage Foundation to mobilize public pressure on behalf of better education for the poor (Carter, 1999). Samuel Casey Carter, a Bradley Fellow at the Heritage Foundation, brought attention to the initiative in his book No Excuses: Lessons from 21 High-Performing, High-Poverty Schools (2000). The philosophy was further heralded by Thernstrom and Thernstrom in their book No Excuses: Closing the Achievement Gap In Learning (2003). The No Excuses philosophy is centered around raising awareness of the idea that there is no excuse for the academic failure of schools serving low-income children and is aimed to embolden educators to develop school-based interventions and schooling models aimed at improving academic outcomes for this population (Carter, 1999). Although the first group of high-performing high-poverty schools recognized for their success using a No Excuses model included public schools, private schools, and charter schools (Carter, 1999), and there is extant literature on the implementation of elements of the No Excuses model in traditional pubic schools (Fryer, 2014), this model of education is currently most often practiced in charter schools. This is likely due to the operational autonomy afforded to charter schools, allowing charter management organizations the freedom to implement innovative models within their schools and making such a model identifiable within the literature. Thus, this review will focus on No Excuses charter schools.

No Excuses charter schools are primarily designed to improve the academic performance of students in the primary and secondary grades, with the aim of minimizing racial and income-based achievement gaps. Several charter networks such as the Knowledge is Power Program (KIPP) have received national attention by espousing a “no excuses” philosophy, and many lesser-known charter schools have adopted a similar model, such as the North Star Academy of Newark, New Jersey, and the Amistad Academy in New Haven, Achievement First, Uncommon Schools (Armor, 2004; Golann, 2015) .

Although a consensus definition of the No Excuses charter school model does not exist, most definitions appear to share five key characteristics: (a) high academic expectations, (b) rigid and consistent discipline, (c) extended instructional time, (d) intensive teacher training, and (e) increased parental involvement (Dobbie & Fryer, 2013; Thernstrom & Thernstrom, 2003; Whitman, 2008; Carter, 2000; Golann, 2015). For the purposes of this review, a charter school will be defined as No Excuses if it employs each of these five key characteristics. However, schools may adopt and employ these characteristics to varying degrees, along with additional policies that may distinguish them from other schools. Additional characteristics outlined by Carter (2000) may include autonomy of administrators, use of measurable goals, employment of master teachers, rigorous and regular testing, and requirement of mastery for grade promotion, among others.

How No Excuses Charter Schools Might Work

As pioneer of culturally relevant pedagogy Gloria Ladson-Billings (2006) conceptualizes the American educational system, she refers to an “educational debt” that in some ways renders more understandable the achievement gap that has been studied over the last thirty years. Her work points to the history of American education as a classist, elitist, and racist one, as she notes that African American, American Indian, and Latino/a peoples were historically excluded from educational opportunities and viewed as uneducable. Even as later attempts at integrating and equalizing the educational system aimed to redress some of this historical wrong, they did so within the same educational frame. With this grounding perspective, it may be easier to understand why some ethnic and racial groups report lower academic achievement on average than their white peers; rather than attributing shortcomings to differences in potential or ability, Ladson-Billings and others urge us to consider the systemic forces that support white students' achievement and limit the achievement of racial and ethnic minority students. The education reformers involved in developing the No Excuses model have attempted to take these forces into consideration in creating a framework for education that counteracts some of the problematic elements of the current educational climate. These key characteristics are described below:

High academic expectations

A common thread in the discussion of the reasons underlying the achievement gap is expectations, as both teacher and parental expectations can influence their students' academic achievement (Fan & Chen, 1999; Jussim, Madon, & Chatman, 1994; Rosenthal & Jacobson, 1968). Education stakeholders have speculated that expectations are reduced for low-income and minority students (Tenenbaum & Ruck, 2007) due to teachers' implicit biases against ethnic minority students (Van den Bergh, Denessen, Hornstra, Voeten, & Holland, 2010) and beliefs that minority students' poor performance stems from innate characteristics rather than situational factors (Jackson, 2002). Specifically, in No Excuses charter schools, teachers are asked to increase expectations of students by benchmarking their students' achievement against that of students in highly successful classrooms across the country (Farr, 2010). As teachers' expectations have their greatest effects in classrooms in which high-achieving and low-achieving students are treated very differently (Kulkinski & Weinstein, 2001), the No Excuses charter school model attempts to address the potential for lower achievement by ensuring that expectations are universally high for all students. Further, Fan and Chen (1999) found a strong relationship between parental expectations and their childrens' academic achievement. Thus, the high academic expectations set forth by the No Excuses charter schools may influence parents' expectations, further enhancing achievement outcomes. Within this model, high academic expectations are often implemented via two pedagogical strategies: (1) a college-preparatory curriculum and (2) curriculum decisions driven by data. Use of college preparatory curricula within traditional school models has shown conflicting results (Allensworth, Nomi, Montgomery, & Lee, 2009; Gamoran & Hannigan, 2000). Data-driven decision making has been associated with significant improvement in student achievement (Carlson, Borman, & Robinson, 2011), however others argue that the existing research on using data to inform instructional decision does not yet provide conclusive evidence of the effects on student achievement and that further research is needed (Hamilton, Halverson, Jaskson, Mandinach, & Supovitz, 2009; Marsh, Pane, Hamilton, 2006).

Rigid and consistent discipline

Rigid and consistent discipline appears to be a common and primary component of many No Excuses charter schools. Rigid disciplinary codes are often reflected in mandated dress codes and highly structured, consistently enforced conduct expectations; setting standards for how students walk in the hallways, demonstrate attentiveness in class, and organize their school work (Goodman, 2013; Lake et al., 2012; Whitman, 2008). Lake and colleagues (2012) demonstrated a positive association between student behavior policies and math and reading achievement in four No Excuses charter school management organizations. Further, consistency of enforcement of behavioral expectations may play an important role in the effectiveness of No Excuses charter schools' disciplinary strategies as disciplinary inconsistency has been associated with behavior problems (Feehan, McGee, Stanton, & Silva, 2011; Stormshak, Bierman, McMahon, & Lengua, 2000). In contrast, research on the effects of school uniforms has been mixed (White, 2000). The disciplinary approach used in the model continues to be controversial. Proponents suggest that the disciplinary approach is necessary to promote self-discipline and personal responsibility in an effort to teach students traditional, middle-class values (Whitman, 2008). Opponents of the disciplinary approach are concerned that the sole focus is compliance—undermining students' self confidence and identities, disallowing students the ability to learn to this for themselves, the rules are too numerous, and students are disciplined through a culture of shame (Ellison, 2012; Goodman, 2013; Lack, 2009) .

Extended instructional time

No Excuses charter schools (and many other school models) make an effort to treat instructional time as an educational resource. Extended time, when used for academic learning, has been shown to exert positive impacts on student achievement (Aronson, Zimmerman, & Carlos, 1999; Farbman, 2012; Farbman & Kaplan, 2005). This extra time may manifest in the form of longer class periods, longer school days, or additional schools days. This additional time may allow teachers to provide students with more individualized instruction or intervention and may result in more overall time spent on task.

Intensive teacher training

Teachers employed at No Excuses charter schools often participate in intensive pre-service training before the beginning of a new school year and ongoing professional development throughout the year in the form of intensive teacher coaching and monitoring. New teachers are observed by master teachers, receive feedback on their performance, and submit lesson plans for review (Lake et al., 2012). Despite several decades of research, the relationship between teacher effectiveness and teacher training has not been conclusively demonstrated for traditional school models. However, Lake and colleagues (2012) found intensive teacher coaching to be an effective strategy for four high-performing No Excuses charter school management organizations.

Parental involvement

The parental accountability implemented in No Excuses charter schools may play a key role in affecting student achievement. Many studies have found that students with involved parents were more likely to earn higher grades and test scores, no matter what their income or background (Fehrman, Keith, & Reimers, 1997; Jeynes, 2012; Steinberg, Lamborn, Dornbusch, & Darling, 1992). Several studies found that families of all income and education levels, and from all ethnic and cultural groups, are engaged in supporting their children's learning at home. White, middle-class families, however, tend to be more involved at school (Henderson & Mapp, 2002). Low-income and minority families, however, often face barriers to sufficient involvement in their children's schooling such as monetary constraints, time constraints, transportation needs, language and other communication differences, and cultural beliefs about the role of family in children's schooling (Boethel, 2003). Critical questions have been raised regarding how parental involvement is conceptualized, especially within low-income and minority communities (Barton, Drake, Perez, St. Louis, George, 2000). Social and cultural contexts must be taken into consideration when defining parental involvement.

The common parental accountability component of the No Excuses charter school model attempts to minimize barriers and cultivate commitment from parents to reinforce school actions with strategies such as written contracts explaining the school's expectations regarding (a) parental responsibilities, (b) academic standards, and (c) conduct and discipline, for example; requiring homework assignments to be checked and signed by parents; parent orientation meetings, and asking that the parent check and sign their child's homework before returning it to the teacher. However, further research is required to provide evidence that parental involvement requirements, such as those employed in a No Excuses charter school, are sufficient for overcoming the barriers noted.

Why it is Important to do this Review

Given the expanded recent interest in No Excuses charter schools as an instrument of education reform for low-income and minority students, now more than ever, there is need for critically evaluated information about their performance. A growing body of evidence suggests that No Excuses charter schools have the potential to generate impressive achievement gains, thus minimizing the racial and income-based achievement gaps (Abdulkadiroglu et al., 2009; Angrist, Pathak, & Walters, 2011; Dobbie & Fryer, 2013). While the results of these studies are encouraging, no high-quality systematic reviews or meta analyses of the evidence currently exist. Thus, such a review is necessary to ensure that this model of schooling precipitates these purported effects, to ensure that students are justly served by its use.

Moreover, as students' poor academic achievement is associated with lower earning potential (Miller, 1998), greater youth violence (Hawkins, Herrenkohl, Farrington, Brewer, Catalono, & Harachi, 1998; Kingery, Pruitt, Heuberger, & Brizzolara, 1996), increased drug use (Cox, Zhang, Johnson, & Bender, 2007; Mensch & Kandel, 1988), and delinquency (Maguin & Leober, 1996), identifying means of decreasing the achievement gap and improving all students' educational performance is an important priority for educational research.

In this review, we intend to evaluate the efficacy of use of the No Excuses charter schools with respect to their impact on students' achievement in math and literacy. To do so, a comprehensive systematic review must be conducted including studies investigating a variety of charter networks implementing the No Excuses charter school model. Taking into account the accumulating literature suggesting the potential benefits of No Excuses charter schools and the lack of prior reviews, there is a need to comprehensively synthesize the full evidence base of methodologically rigorous studies that examine impacts of No Excuses charter schools on math and literacy achievement. A systematic review that evaluates the efficacy of the model will advance the evidence base and provide researchers, educators, and policymakers with a means by which to make informed decisions about the use of the model in the field.

Logic model

Conceptual understanding of the causal pathways through which No Excuses charter schools may influence students' math and literacy achievement is currently unclear. It was necessary, therefore, to develop a logic model specifically for this review, reflecting the purported pedagogical mechanisms discussed above. The logic model illustrates both intermediary factors through which No Excuses charter schools may exert their impact on achievement measures, and additional contextual factors that may modify or inhibit the desired effect. An initial version of this model is detailed in Figure 1.

Preliminary Logic Model for No Excuses Charter Schools as an Intervention for Students' Math and Literacy Outcomes

OBJECTIVES

The systematic review and meta-analysis aims to examine the available evidence on the impacts of No Excuses charter schools on students' achievement in math and literacy outcomes, relative to students enrolled in traditional public schools. The focal research questions are as follows:

Does the No Excuses charter school model effectively enhance students' math or literacy achievement? What is the magnitude and variability of effects?

How do these effects translate to changes in racial and income-based achievement gaps?

What are the gaps in the literature and limitations to the evidence?

Secondarily, should the included studies provide us with sufficient relevant data to explore differential effects for participants with different characteristics (i.e., potential moderators), we will explore their presence and magnitude.

METHODOLOGY

Criteria for including and excluding studies

A study must meet all criteria set forth below to be eligible for inclusion in this systematic review and meta-analysis.

Types of study designs

Studies must use an experimental randomized controlled trial (RCT) or controlled quasi-experimental design (QED). Because RCTs can control for confounding variables, such as parental involvement or student motivation, that might be causally related to differential outcomes in achievement between No Excuses charter and comparison public schools, RCTs are preferred to QEDs. For QED studies to be eligible, researchers must compare No Excuses charter school students to demographically similar students who attend traditional public schools. Methods used to establish the demographic similarity of groups must include some measure of past achievement, but measures of race, ethnicity, and socioeconomic status are also desired. Appropriate methods of establishing comparable groups include: individual or school-level matching; regression models that use student-fixed-effects to adjust for student characteristics; or instrumental variables analyses that adjust for an instrument for whether students enroll in a charter school.

Eligible QED studies must report or control for baseline data on some achievement measure separately for the intervention and comparison conditions. If baseline data are not explicitly provided (e.g., means and standard deviations of prior achievement for each condition), then the authors must use baseline data to statistically adjust for differences in prior achievement in their analyses (e.g., use prior achievement as a covariate in a regression model).

Types of participants

This review will specifically examine students enrolled in primary and secondary No Excuses charter schools. The student populations at No Excuses charter schools are predominantly low-income and/or minority.

Types of interventions

This review will examine charter schools that implement a No Excuses model, both in the U.S. and abroad. To determine if a charter school uses a No Excuses model, it will be evaluated for the presence of five key characteristics discussed above (i.e., (a) high academic expectations, (b) rigid and consistent discipline, (c) extended instructional time, (d) intensive teacher training, and (e) increased parental involvement). All five key characteristics must be present for a school to be identified as a No Excuses charter school. These characteristics must be mentioned explicitly as being endorsed by the charter school, although leniency will be granted for the degree of operationalization (e.g., what is “rigid and consistent” discipline and “intensive” teacher training). This method of defining a No Excuses charter school should include schools that unknowingly conform to the No Excuses model without including charter schools that only employ part of the No Excuses model. Because our interest is in No Excuses charter schools, studies that compare achievement in public schools to charter schools, without adequate information to separate No Excuses charter schools from charter schools emplying other models, will not be eligible.

Types of outcome measures

Eligible studies must measure at least one standardized student-level math or literacy achievement outcome. All meta-analyses will be conducted separately for the math and literacy outcomes (described in more detail in the synthesis section below).

Math and literacy outcomes standardized to national, state, or another representative samples will be used. These can be defined as standardized assessments on which student demonstrate grade-level appropriate knowledge of math or literacy domains. For example, math or literacy subtests of the NEAP, SAT, ACT, PISA are all acceptable standardized measures. Raw scores, standardized scores, and percentile scores are all acceptable outcomes. All unstandardized outcomes, such as curriculum-based measurements, grade point averages, or teacher-reported grades, will be excluded from the review to minimize potential bias from such unstandardized assessments. Because outcomes are standardized achievement measures (where, to our knowledge, higher scores always indicate higher achievement), combining effects from different scales will not be an issue.

Studies that only combine math and literacy outcomes into a unified achievement measure will not be included in the review. This exclusion is because we will be evaluating the impact of the intervention on math and literacy achievement separately, and are not interested in examining effects on broader measures of achievement that span both math and literacy domains.

Studies must have been conducted after 1990. This is the year the first charter school was founded, and thus we do not expect any studies to have been conducted prior to this date.

Eligible studies may have been conducted in any country. However, the cultural factors informing the development and implementation of No Excuses charter schools may largely be limited to the heterogeneous population of the United States. Consequently, the country of origin in which the research was conducted will be examined as a moderator in analysis (described later).

To be included in the quantitative synthesis, studies must report statistics necessary for calculating effect sizes. If multiple forms of an eligible outcome are reported (such as means and standard deviations as well as pass rates), we will use the maximally informative form. When possible, we will solicit authors for missing information, or information on an eligible outcome that is reported in an unusable way, to calculate effect sizes by hand. However, studies ultimately missing effect size information will still be included in the systematic review to summarize characteristics about No Excuses charter schools and their students.

Duration of follow up

We expect most eligible studies will only assess achievement annually (i.e., few studies will report monthly or mid-year results). Although one-year follow-ups post-assignment are expected to be the most common duration, we anticipate some eligible studies that follow up samples longitudinally with annual assessments. As such, we will conduct separate meta analyses for each yearly follow up (i.e., first year follow up, second year follow up).

Types of settings

Studies conducted in natural school settings are eligible for inclusion in this review. Studies conducted in non-school settings are ineligible due to external validity concerns including the potential limitation in providing generalizability of findings to the target population.

Search strategy

We will conduct a comprehensive systematic literature review to identify all eligible studies, regardless of publication status. The search will be conducted using electronic databases, internet search engines, citations in previous meta-analyses and literature reviews, citations in research reports screened for eligibility, conference listings, hand searches of relevant journals, and correspondence with experts in the field. Redactions and erratum for potentially eligible studies will be screened. We will use date-of-publication restrictions limiting our search years to 1990 or later as No Excuses charter schools did not exist until the mid-1990s. We will repeat the search process before publication to ensure any recently published and eligible studies have been included in the review and final analyses.

Electronic Searches

We will use the ProQuest database host to search 15 electronic databases:

PsycINFO, PsycARTICLES, ProQuest Social Sciences Premium Collection (includes ERIC), ProQuest Dissertations & Theses Global, ProQuest Sociology, ProQuest Social Science Journals, ProQuest Science Journals, ProQuest Religion, ProQuest Psychology Journals, ProQuest Political Science, ProQuest Health and Medical Complete, ProQuest Family Health, ProQuest Education Journals, ProQuest Criminal Justice, PILOTS, and ebrary e-books.

Table 1

Preliminary Search Strategy in ProQuest Search Description

Search Description
Limited to title and abstract	ti,ab(“charter school” OR “charter schools” OR charter* OR “KIPP” OR “Achievement First” OR “Uncommon schools” OR “Match Education” OR “Friendship public” OR “Breakthrough Schools” OR “Brighter Choice” OR “IDEA public” OR “YES prep” OR ”Alliance of College Ready“ OR ”Alliance College-Ready“ OR ”Rocketship Education“ OR ”Summit Public“ OR ”Green Dot“ OR ”Great Hearts“ OR ”STRIVE Prep“ OR ”Strive Preparatory“ OR ”Aspire Public“ OR ”Bright Star“ OR ”Leadership Public“ OR ”Success Academies“ OR ”Summit public“ OR ”Synergy Academies“ OR ((foundation OR free OR paternal* OR autonom) PRE/1 school) OR ((”academy“ OR ”academies“) AND (”United Kingdom“ OR ”England“ OR ”British“ OR ”Britain“))) AND ti,ab(”lottery“ OR compar* OR random* OR control* OR propensity OR match* OR experiment) AND ti,ab(achiev OR assess* OR math* OR liter* OR read* OR effect* OR outcome* OR standard* OR affect* OR impact*)
Excluded from title	NOT ti(lunch* OR meal* OR breakfast* OR drug* OR smok* OR bully* OR psychiatr*)
Additional delimiters	Date: After 1 January 1990

The tentative search strategy that will be used to search ProQuest is outlined in Table 1. Beyond including charter and charter schools in the search, we will include specific brands of charter schools in the United States, such as KIPP, Rocketship Education, and Great Hearts. To identify international studies, we will use the search terms free, paternalistic, autonomous, foundation, and academies which must precede school or schools. Through extensive scoping searches, we found these to be the most common terms for charter schools internationally. We suspect the term free school will draw a particularly diverse swath of irrelevant studies, including literature related to smoke-free schools, free school meals, and the Free School Movement.

To ensure comprehensive inclusion of relevant studies, we will limit search terms about design and outcomes to only the broadest terms. While we hope to target well-described studies with terms such as lottery, propensity, or experiment*, we also will include terms such as match*, compar*, and control* to target studies that do not describe their methodology in great detail. To search eligible standardized achievement outcomes, we will search standard*, achiev*, and assess* as well as effect*, affect, impact* and outcome*.

Searching other resources

In addition to the electronic search of the above databases, we will review the references lists of all retrieved documents. We will also hand-search the tables of contents of key journals: (e.g., Educational Evaluation and Policy Analysis). The websites of relevant professional organization such as Mathematica, RAND, and AIR will be searched for potentially eligible research reports. Finally, key investigators who are known to be active in the field and experts identified during the review process will be contacted with a request to share any published, unpublished, and ongoing research relevant to the review.

Selection of studies

Two reviewers will independently review the titles and abstracts for all identified studies. Any studies deemed irrelevant at this stage by both reviewers will be discarded. Any studies that are deemed potentially eligible at this stage by at least one reviewer will be retrieved in full text. Studies retrieved in full-text screening will then be screened for eligibility by two reviewers. Any disagreements about eligibility will be resolved via discussion with a third review author. We will document all reasons for exclusions at the full text level, which will be reported in the study flow diagram in the completed review.

Data extraction and management

Two reviewers will independently extract data from at least 30% of the eligible studies. Periodic reliability assessments to protect against coder drift will be conducted during the initial stages of coding. If a study is found to be ineligible at the full-text coding stage, the reasons for ineligibility will be documented and reported. Disagreements in coding will be resolved through discussion and consensus; any disagreements about coding items will be resolved through discussion with the third review author. Before synthesis of results, effect sizes will be assessed for numerical accuracy to ensure extracted effect sizes match the direction of effects reported.

Prior to coding, the authors will participate in comprehensive training involving review of coding rules and areas of ambiguity in the coding manual. Inter-rater reliability checks will be conducted prior to coding the final set of eligible studies. Official coding will not begin until all coders achieve at least 80% agreement on a set of five randomly selected eligible research reports. Once the criterion for inter-rater agreement is met, any disagreements on codes of the five studies will be discussed and a consensus code will be used. Studies that meet reliability standards will be used as part of the final data. Data extraction forms for reliability, coding, and screening will be developed in EndNote.

To ensure robustness of findings, several study and design characteristics have been pre-identified for inclusion in moderator analyses. Study design (RCT vs. QED) and study source (e.g., journal, dissertation, unpublished study) will be included to determine if there are significant differences between randomized and non-randomized findings and to determine if unpublished reports—commonly sponsored by charter organizations—have significantly different findings from peer-reviewed published articles or dissertations. The types of reported academic measures will also be included, separating state-standardized from nationally-standardized tests, to determine if findings differ for measures which vary widely in difficulty from state to state.

Student gender and grade level will also be included as moderator variables to determine if effects differ for boys and girls, as the high-expectations and strict discipline within No Excuses charter schools may help male students' performance more than female students'. Grade level will also be included to serve as a proxy for exposure length. It may be important to evaluate 9^th-grade achievement scores separately from 12^th-grade test scores, as older students will likely have had more exposure to the No Excuses charter school model. Finally, because No Excuses schools draw from low-income and/or minority communities, analyzing race or socio-economic status as moderator variables will likely not be meaningful.

School level variables such as class size, school size, teacher demography, and age of charter network will also serve as important moderator variables. By reporting separate effect sizes for class and school size, the researchers hope to tease out what aspects of achievement are less related to the No Excuses charter school model and more related to school-level variables that may be malleable for traditional public schools. Moderator analyses will also be conducted by teacher demography to determine if teacher racial or gender differences are associated with different effects of the intervention. Charter school-specific moderators such as the age of the charter network will provide important insight into the differences within No Excuses charters. It is important to differentiate between charters that were recently founded (less than five years before measurement) and charters that are more established so that the researchers can better understand the role experience plays in achievement effects. Finally, understanding whether the No Excuses charter is part of a larger network or is a standalone, start-up charter will give insight into the importance of organizational support and resources.

Assessment of risk of bias in included studies

In order to obtain robust and meaningful results, we will conduct a thorough assessment of the risk of bias of all studies included in the meta-analysis. For RCTs, we will employ the Cochrane risk-of-bias tool, with particular attention to the following domains: incomplete outcome data, selective reporting bias, or other risk of bias. We do not anticipate sequence generation, allocation concealment, and blinding to vary in this literature for RCTs, because RCTs will almost ubiquitously employ lottery assignment to condition, treatment providers are teachers (who cannot be blind to their respective condition), and standardized achievement measures (which are often scored blindly by design). While we do not anticipate studies to selectively report outcomes (such as reporting favorable results for a standardized math outcome but failing to report the accompanying literacy outcome), we will assess selective reporting bias and report on any suspected bias in the review, if found. Results from the Cochrane risk-of-bias tool will be summarized for included RCTs. Additional direct coding will be employed for all studies (including QEDs) with regard to selection bias, overall attrition, differential attrition between groups, and baseline equivalence or pretest effect sizes indexing differences between groups on key characteristics.

Description of methods used in primary research

We expect QEDs to be more common among included studies than RCTs because lottery assignment can only occur at over-subscribed charter schools. We expect most eligible studies will compare only one No Excuses charter school to public school students at either one school or at public schools across a school district, although other comparisons (such as multiple No Excuses charter schools managed by the same charter management organization compared to multiple public schools) are possible. We also expect most eligible studies will only assess achievement once (i.e. few studies will report follow-up results), approximately one year after lottery allocation or matching, because most standardized assessments occur annually. However, we do anticipate some studies will include longitudinal results, and thus report on achievement measures annually for several years after lottery allocation or matching.

Criteria for the determination of independent findings

We will take several steps to ensure that all effect sizes used in any given meta-analysis represent statistically independent observations. First, we will collate multiple reports on a single sample into one study. Second, if multiple time points are reported for an eligible study, we will use follow ups in separate meta-analyses as described above. Finally, if a study reports multiple math measures or multiple literacy measures (respectively) we will average the measures into a single outcome within that outcome domain using the procedure outlined above (Borenstein et al., 2009 p.230).

Details of study coding categories

All studies that meet the eligibility criteria in the citation and abstract, and full-text screening will be coded using a detailed coding instrument developed by the authors and modeled on codebooks developed by Tanner-Smith & Lipsey (2009). Data extracted from primary studies will include information on methodology, design, intervention and control group characteristics, math or literacy outcome details and characteristics, etc.

Statistical procedures and conventions

Measures of treatment effect

We expect most studies will report continuous measures of math or literacy outcomes, so will use the small-sample corrected standardized mean difference effect size metric (Hedges' g) to estimate the mean difference in outcomes by intervention and comparison conditions. If an eligible study reports only binary outcomes (e.g., pass-fail rates for an achievement test) by condition, we will compute log odds ratios; we will then convert these log odds ratios to Hedges' g effects (using procedures outlined in Borenstein et al., 2009, p. 47) and synthesize these effect sizes in a single meta-analysis.

After converting all estimated mean differences to Hedges' g, outliers will be defined as those effect sizes more extreme than three times the standard error of the mean effect size from the mean effect size. Once defined as an outlier, the effect size will be Winzorized to the next closest (non-outlier) estimate.

Unit of analysis issues

We do not expect studies in this literature to use cluster randomized or cluster quasi-experimental designs. In the event that studies do use cluster level assignment, we will adjust the standard errors of all effect size estimates using the method described in the Cochrane Handbook (Higgins, 2008). If the intraclass correlation (ICC) needed to make this adjustment is not reported in the primary studies, we will use similar ICCs reported in other education trials (Hedges & Hedberg, 2007) and conduct sensitivity analyses using a range of plausible values.

Eligible studies must use student-level outcomes, even if group-wise matching was implemented. Although school-level achievement measures might be common for general comparison of charter and traditional public schools, we anticipate that nearly all studies investigating only No Excuses charter schools (and their traditional public school counterparts) will conform to this format.

Assessment of heterogeneity

We expect some degree of heterogeneity in the included studies, given diversity in participant populations, charter school approaches, study designs, and outcome characteristics. We will therefore use Q, τ², and I² to assess heterogeneity.

Assessment of reporting biases

Publication bias stems from failing to detect unpublished studies. Underrepresentation of unpublished studies, which are more likely to have non-significant effects, can substantially bias effect size estimates (Borenstein et al., 2009). Although all reasonable attempts will be made to include unpublished research such as searching databases of conference proceedings and other grey literature and corresponding with experts in the field, publication bias is always a potential threat to the validity of a meta-analysis. Potential publication bias will therefore be examined using Egger's regression-based assessment of asymmetry of funnel plots (Egger, Smith, Schneider, & Minder, 1997). If publication bias is suggested by the Egger test, then the Duval and Tweedie's (2000) trim and fill method will be used to estimate the potential impact of publication bias on the mean effect size estimates.

Data synthesis

Given the presumed heterogeneity in included studies, we will use random-effects meta analysis models to synthesize findings across studies. All analyses will be conducted separately for the two outcomes: math and literacy. All analyses will be conducted separately for RCTs and QEDs, unless no difference between groups is found, in which case we will instead use a sensitivity analysis approach to synthesize all studies, for which we would statistically adjust for design type in all analyses using meta-regression models. Finally, as noted above, all analyses will also be split by follow-up timing of the outcome measurement. All analyses will be performed using meta-analysis commands that run in the Stata statistical environment (StataCorp, 2011).

Moderator and subgroup analysis

For each subset of studies (i.e., outcome type by study design by follow-up time), we will conduct mixed-effects meta-regression models to explore the potential effect of the following moderator variables: percent minority, percent female, average family income, publication type, years the charter school opened, and type of charter (managed by CMO or standalone). Risk of bias (high versus low) for RCTs will also be included as a moderator to assess sensitivity of results to study bias. These moderator analyses will only be estimated when at least five studies contribute to a given subset. Estimates of the residual Cochrane's Q, I² , and τ² will be used to assess residual variability in the effect sizes after inclusion of the potential moderators; and we will examine the magnitude and statistical significance of the slope coefficients from the meta-regression models to assess moderator effects. Further, we will conduct subgroup analyses by gender (male and female) and ethnicity (white versus non-white) if at least two eligible studies report achievement results by subgroups.

Summary characteristics

We will summarize characteristics of the included studies on No Excuses charter schools and their students using the following table.

Type	Summary Information
Assignment
	RCT
	QED
Publication Type
	Journal
	Thesis
	Report
	Working Paper
	Type of Charter
	CMO
	Standalone
Charter Characteristics		Range	Median
	Grade range
	% minority
	% male
	% low income
	School size
	Total charter N
	Median total N

Sensitivity analyses

Several types of sensitivity checks will be conducted to investigate the robustness of the results. Sensitivity analyses will be used to examine how sensitive results are to 1) Winsorized outliers and 2) different plausible values for ICC estimates for studies using cluster assignment. The potential impact of results from the sensitivity analysis on the findings will be addressed in the final review. We will also conduct a sensitivity analysis that uses the risk of bias scores as moderator variables to examine whether effect estimates are systematically different in studies with different risks of bias.

Footnotes

APPENDIX: CODING MANUAL

References

The Achievement Gap Initiative at Harvard. (2013). Facts on achievement gaps. Retrieved from http://www.agi.harvard.edu/projects/FactsonAchievementGaps.pdf.

Allensworth

Elaine

Nomi

Takako

Montgomery

Nicholas

, and Lee

Valerie E.

. “College preparatory curriculum for all: Academic consequences of requiring algebra and English I for ninth graders in Chicago.” Educational Evaluation and Policy Analysis 31, no. 4 (2009): 367–391.

Armor

(2004). No excuses: Simplistic solution for the achievement gap? The Teachers College Record. Retrieved December 19, 2014, from http://www.tcrecord.org/content.asp?contentid=11268.

Armor

Watkins

(2006). School segregation and black achievement: New evidence from the 2003 NAEP. The benefits of racial and ethnic diversity in elementary and secondary education, 28–49.

Aronson

Zimmerman

Carlos

(1999). Improving Student Achievement by Extending School: Is It Just a Matter of Time?.

Barton

Coley

(2007). The family: America's smallest school. Princeton, NJ: ETS Policy Information Center.

Berends

Penaloza

(2010). Increasing racial isolation and test score gaps in mathematics: A 30-year perspective. The Teachers College Record, 112(4), 2–3.

Boethel

Diversity: School, family, & community connections.

SEDL Annual Synthesis, 2003.

Borenstein

Hedges

L. V.

Higgins

Rothstein

H. R.

(2009). Introduction to meta analysis. Chichester: John Wiley & Sons, Ltd.

10.

Caldas

S. J.

Bankston

(1998). The inequality of separation: Racial composition of schools and academic achievement. Educational Administration Quarterly, 34(4), 533–557. doi:10.1177/0013161X98o34004005

11.

Carlson

Borman

G. D.

Robinson

(2011). A multistate district-level cluster randomized trial of the impact of data-driven reform on reading and mathematics achievement. Educational Evaluation and Policy Analysis, 33(3), 378–398.

12.

Carter

S. C.

(1999). No excuses: Seven principals of low-income schools who set the standard for high achievement (Descriptive Report No. 141). Heritage Foundation, Washington, DC.

13.

Carter

S. C.

(2000). No excuses: Lessons from 21 high-performing, high-poverty schools. Heritage Foundation, 214 Massachusetts Avenue, NE, Washington, DC 20002.

14.

Cox

R. G.

Zhang

Johnson

W. D.

Bender

D. R.

(2007). Academic performance and substance use: findings from a state survey of public high school students. Journal of School Health, 77(3), 109–115.

15.

Dobbie

Fryer

R. G.

, (2013). Getting beneath the veil of effective schools: Evidence from New York City. American Economic Journal: Applied Economics, 5(4), 28–60. doi: 10.1257/app.5.4.28.

16.

Duncan

Magnuson

(2013). The long reach of early childhood poverty. In Yeung

W.-J. J.

Yap

M. T.

(Eds.), Economic Stress, Human Capital, and Families in Asia (Vol. 4, pp. 57–70): Springer Netherlands. doi:10.1007/978-94-007-7386-8_4

17.

Ellison

(2012). It's in the name: A synthetic inquiry of the Knowledge is Power Program [KIPP]. Educational Studies: A Journal of the American Educational Studies, 48(6), 550–575. doi: 10.1080/00131946.2011.647156

18.

Fan

Chen

(2001). Parental involvement and students' academic achievement: A meta-analysis. Educational psychology review, 13(1), 1–22.

19.

Farbman

(2012). The Case for Improving and Expanding Time in School: A Review of Key Research and Practice. National Center on Time & Learning.

20.

Farbman

Kaplan

(2005). Time for a Change: The Promise of Extended-Time Schools for Promoting Student Achievement. Research Report. Massachusetts 2020.

21.

Feehan

McGee

Stanton

W. R.

Silva

P. A.

(1991). Strict and inconsistent discipline in childhood: consequences for adolescent mental health. British journal of clinical psychology, 30(4), 325–331.

22.

Fehrmann

P. G.

Keith

T. Z.

Reimers

T. M.

(1987). Home influence on school learning: Direct and indirect effects of parental involvement on high school grades. The Journal of Educational Research, 80(6), 330–337. doi:10.1080/00220671.1987.10885778

23.

Fiel

J. E.

(2013). Decomposing school resegregation social closure, racial imbalance, and racial isolation. American Sociological Review, 78(5), 828–848. doi:10.1177/0003122413496252

24.

Fryer

R.G.

, and Levitt

S.D.

(2004). Understanding the black-white test score gap in the first two years of school. Review of Economics and Statistics, 86, 447–464. doi:10.1162/003465304323031049

25.

Gamoran

Hannigan

E. C.

(2000). Algebra for everyone? Benefits of college-preparatory mathematics for students with diverse abilities in early secondary school. Educational Evaluation and Policy Analysis, 22(3), 241–254.

26.

Golann

J. W.

(2015). The paradox of success at a no-excuses school. Sociology of Education, 88(2), 103–119. doi: 10.1177/0038040714567866

27.

Goodman

J. F.

(2013). Charter management organizations and the regulated environment: Is it worth the price? Educational Researcher, 42(2), 89–96. doi: 10.3102/0013189x12470856

28.

Hamilton

Halverson

Jackson

S. S.

Mandinach

Supovitz

J. A.

Wayman

J. C.

, … & Steele

J. L.

(2009). Using student achievement data to support instructional decision making.

29.

Hanushek

E. A.

Rivkin

S. G.

(2009). Harming the best: How schools affect the black-white achievement gap. Journal of Policy Analysis and Management, 28(3), 366–393. doi: 10.1002/pam.20437

30.

Hawkins

J.D.

Herrenkohl

Farrington

D. P.

Brewer

Catalano

R. F.

Harachi

T.W.

(1998). A review of predictors of youth violence. In Loeber

and Farrington

D.P.

(Eds.), Serious and Violent Juvenile Offenders: Risk Factors and Successful Interventions, (pp. 106–146). Thousand Oaks, CA: Sage Publications.

31.

Henderson

A. T.

Mapp

K. L.

A new wave of evidence: The impact of school, family, and community connections on student achievement.

SEDL Annual Synthesis, 2002.

32.

Jacob

Ludwig

(2008, December) Improving educational outcomes for poor children. (NBER Working Paper No. W14550). doi: 10.3386/W14550

33.

Jackson

S.A.

(2002). A study of teachers' perceptions of youth problems. Journal of Youth Behavior, 5, 313–322.

34.

Jeynes

(2012). A meta-analysis of the efficacy of different types of parental involvement programs for urban students. Urban Education, 47(4), 706–742. doi:10.1177/0042085912445643

35.

Jencks

Mayer

S. E.

(1990). The social consequences of growing up in a poor neighborhood. In Lynn

L. E.

McGeary

M. G.

(Eds.), Inner-City Poverty in the United States, (pp. 111–186): National Academies. doi: 10.17226/1539

36.

Jussim

Madon

Chatman

(1994). Teacher expectations and student achievement. In Applications of heuristics and biases to social issues (pp. 303–334). Springer US.

37.

Kingery

P.M.

Pruitt

B.E.

Heuberger

Brizzolara

J.A.

(1996). Violence prevention in rural areas: Evidence of the need for educational reform and community action. International Journal of Educational Reform, 5(1), 26–34.

38.

Kuklinski

M.R.

Weinstein

R.S.

(1996). Classroom and developmental differences in a path model of teacher expectancy effects. Child Development. 2001(72), 1554–1578.

39.

Lack

(2009). No excuses: A critique of the Knowledge is Power Program (KIPP) within charter schools in the USA. Journal for Critical Education Policy Studies, 7(2), 126–153.

40.

Ladson-Billings

(2006). From the achievement gap to the education debt: Understanding achievement in US schools. Educational researcher, 35(7), 3–12.

41.

Lake

Bowen

Demeritt

McCullough

Haimson

Gill

(2012). Learning from Charter School Management Organizations: Strategies for Student Behavior and Teacher Coaching. Mathematica Policy Research, Inc.

42.

Maguin

Loeber

(1996). Academic performance and delinquency. Crime and justice, 145–264.

43.

Marsh

J. A.

Pane

J. F.

Hamilton

L. S.

(2006). Making sense of data-driven decision making in education.

44.

Mensch

B. S.

Kandel

D. B.

(1988). Dropping out of high school and drug involvement. Sociology of Education, 95–113.

45.

Miller

S. R.

(1998). Shortcut: High school grades as a signal of human capital. Educational Evaluation and Policy Analysis, 20(4), 299–311.

46.

National Poverty Center, University of Michigan (2010). Retrieved on October 24, 2015 from http://npc.umich.edu/poverty/.

47.

Reardon

S. F.

(2008). Thirteen ways of looking at the black-white test score gap. (Working Paper No. 2008-8). Retrieved from Stanford Institute for Research on Education Policy and Practice website: http://www.computing.ccpr.ucla.edu/events/ccpr-previous-seminars/ccpr-seminars-previous-years/Reardon_13_ways.pdf.

48.

Reardon

S. F.

(2013). The widening income achievement gap. Educational Leadership, 70(8), 10–16.

49.

Rosenthal

Jacobson

(1968). Pygmalion in the classroom: Teachers' expectations and pupils' intellectual development. New York: Rineholt and Winston.

50.

Rumberger

Palardy

(2005). Does segregation still matter? The impact of student composition on academic achievement in high school. The Teachers College Record, 107(9), 1999–2045.

51.

Steinberg

Lamborn

S. D.

Dornbusch

S. M.

Darling

(1992). Impact of parenting practices on adolescent achievement: Authoritative parenting, school involvement, and encouragement to succeed. Child development, 63(5), 1266–1281. doi:10.1111/j.1467-8624.1992.tbo1694.x

52.

Stiefel

Schwartz

A. E.

Gould Ellen

(2007). Disentangling the racial test score gap: Probing the evidence in a large urban school district. Journal of Policy Analysis and Management, 26(1), 7–30. doi:10.1002/pam.20225

53.

Stormshak

E. A.

Bierman

K. L.

McMahon

R. J.

Lengua

L. J.

(2000). Parenting practices and child disruptive behavior problems in early elementary school. Journal of clinical child psychology, 29(1), 17–29.

54.

Tanner-Smith

E. E.

Lipsey

M. W.

(2009). Brief intervention effects on adolescent and young adult's alcohol consumption. Unpublished meta-analysis coding manual. Vanderbilt University, Nashville, TN.

55.

Thernstrom

Abigail M.

and Thernstrom

Stephan

. 2003. No Excuses: Closing the Racial Gap in Learning. New York: Simon & Schuster. doi: doi.org/10.1086/424723

56.

Tuttle

C. C.

Gill

Gleason

Knechtel

Nichols-Barrer

Resch

(2013). KIPP Middle Schools: Impacts on Achievement and Other Outcomes (Final Report # 06441.910). Retreived from Mathematica Policy Research, Inc. website: http://www.mathematica-mpr.Com/∼/media/publications/PDFs/education/KIPP_middle.pdf

57.

U.S. Census Bureau . (2010). U.S. Census Project. Retrieved from http://www.census.gov/newsroom/releases/archives/population/cb12-243.html.

58.

U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. (2014). National Assessment of Educational Progress-Long Term Trend Assessment. Washington, DC: Authors.

59.

Van den Bergh

Denessen

Hornstra

Voeten

Holland

R. W.

(2010). The implicit prejudiced attitudes of teachers relations to teacher expectations and the ethnic achievement gap. American Educational Research Journal, 47(2), 497–527.

60.

White

K. A.

(2000). Do School Uniforms Fit?. School Administrator, 57(2), 36–40.

61.

Whitman

(2008). Sweating the small stuff: Inner-city schools and the new paternalism. Washington, DC: Thomas B. Fordham Institute.