Targeted school-based interventions for improving reading and mathematics for students with,or at risk of,academic difficulties in Grades 7–12: A systematic review

Theoretical perspectives

The reasons why students may be struggling are multifaceted and the theoretical perspectives underlying interventions are therefore likely to be broad. Nevertheless, three superordinate components are characteristic for the majority of the included interventions. These components can be abridged to:

Adaptation of behaviour (social learning theory).

We emphasise that the following presentation of theoretical perspectives is not all-encompassing and although components are presented as demarcated, they contain some conceptual overlap.

Social learning theory has its origins in social and personality psychology and was initially developed by psychologist Julian Rotter and further developed especially by Bandura (1977; 1986). From the perspective of social learning theory, behaviour and skills are primarily learned by observing and imitating the actions of others, and behaviour is in turn regulated by the recognition of those actions by others (reinforcement) or discouraged by a lack of recognition or sanctions (punishment). According to social learning theory, creating the right social context for the student can therefore stimulate more productive behaviour through social modelling and reinforcement of certain behaviours that can lead to higher achievement.

Cognitive developmental theory is not one particular theory, but rather a myriad of theories about human development that focus on how cognitive functions such as language skills, comprehension, memory and problem-solving skills enable students to think, act and learn in their social environment. Some theories emphasise a concept of intelligence where children gradually come to acquire, construct, and use cognitive functions as the child naturally matures with age (e.g., Perry, 1999; Piaget, 2001). Other theories hold a more socio-cultural view of cognitive development and use a more culturally distinct and individualised concept of intelligence that, to a greater extent, includes social interaction and individual experience as the basis for cognitive development. Examples include the theories of Sternberg (2009) and Gardner (1999).

Pedagogical theory draws on different disciplines in psychology and social theory such as cognitivism, social-interactional theory and socio-cultural theory of learning and development. There is not one uniform pedagogical model, but examples of contemporary models in mainstream pedagogy are concepts such as Scaffolding (Bruner, 2006) and the Zone of Proximal Development (Vygotsky, 1978), which originate in developmental and educational psychology. These theoretical positions hold that learning and development emerge through practical activity and interaction. Acquisition of new knowledge is therefore considered to be dependent on social experience and previous learning, as well as the availability and type of instruction. Accordingly, school interventions require educators to interact and organise the learning environment for the student in certain ways to fit the individual student's needs and potential for development.

Interventions in practice

School interventions affect academic achievement either by changing the methods by which instruction is given (the instructional method), or by changing the content of what is taught (the content domain). Many interventions combine several programmatic components as well as theoretical perspectives. Examples from included interventions of instructional methods are small group instruction, coaching of teachers, peer-assisted instruction, CAI, incentives and increased progress monitoring. Included reading interventions target content domains such as comprehension, fluency, decoding and vocabulary (most targeted more than one reading domain). Mathematics interventions target the following domains: operations (e.g., addition, subtraction and multiplication), word problems, fractions, algebra and general math proficiency (or multiple components). Some interventions target both mathematics and reading, and some target other types of skills, such as social-emotional skills or meta-cognitive strategies (e.g., learning how to learn, or self-regulation strategies).

Earlier research indicates that different types of interventions can improve the academic achievement of students with or at risk of difficulties, both across methods, delivery mode, age group and duration (e.g., Cheung & Slavin, 2012; Dietrichson et al., 2017; Gersten et al., 2009; Slavin & Madden, 2011). For example, both reading strategy instruction and peer-mediated learning programmes such as paired reading have shown positive effects on the literacy skills of struggling secondary school readers. These are two types of programmes that clearly have different components and delivery modes (Edmonds et al. 2009). In another example, Good et al. (2003) showed that changing expectations of seventh grade students at risk for stereotype-based underperformance (minority and low-income students in general, and girls regarding mathematics) can improve standardised test scores.

Furthermore, interventions, such as tutoring and structured peer-assisted interventions, often have in common that they comprise an eclectic theoretical model that combines components from all three perspectives on learning presented in the previous section. They are comprehensive interventions that rely on a complexity of mechanisms such as increased feedback and tailor-made instruction (pedagogical theory), regulation of behaviour by, for example, rewards or close interaction with role models (social learning theory), and development of meta-cognitive functions such as learning how to learn (cognitive developmental theory).

Another way of viewing these and other types of interventions is that they attempt to address the differential family and neighbourhood resources of students with high and low risk of academic difficulties. Low risk students are more likely to have access to “tutors” all year round, in the form of parents, siblings and other family members who help out with homework and schoolwork. Interventions to change mindsets, improve expectations, and mitigate stereotype threat also target areas where low risk students may have an advantage, as low risk students are less likely to be subject to stereotype threat, and their families and teachers may already be teaching their children growth mindsets and have favourable expectations of their academic achievement. Different types of extrinsic rewards may be a way to bolster motivation, which may be especially important for students whose families place less weight on educational achievement.

Furthermore, if, as indicated in the previous section, the differences between high and low risk students can be understood as a consequence of differential access to a combination of resources, remedial efforts may need to address several problems at once in order to be effective. Programmes that combine certain components may therefore be more effective than others. Another reason to examine combinations of components relates to an often-suggested explanation for missing impacts: lack of motivation among participants (e.g., Edmonds et al. 2009). It is therefore possible that interventions will be more effective if they also include some form of reward for participating students and implementing teachers, along with other components providing, for instance, pedagogical support.

For struggling students in Grades 7–12 (ages range from 12–14 to 17–19, depending on country), who are likely to have a history of low achievement, finding the right combination of intervention components may be especially pertinent (e.g., Edmonds et al. 2009). Some researchers have recommended, based on the perceived low relative cost-effectiveness of interventions directed to adolescents, that resources should disproportionally be used for early interventions (e.g., Esping-Andersen, 2004, Heckman, 2006), or that secondary schools should primarily be providing technical and vocational training for disadvantaged teenagers (Cullen, Levitt, Robertson, & Sadoff, 2013). However, Cook et al. (2014) argued that the low cost-effectiveness may be a premature conjecture, as previous interventions for youths have often not combined the fostering of academic skills with other important factors for academic success, such as social-cognitive skills. As, for example, social information processing programmes (Wilson & Lipsey, 2006a; 2006b), and programmes based on cognitive behavioural therapy (e.g., Lipsey, Landenberger, & Wilson, 2007) have been found to effectively reduce problematic behaviour and promote social-cognitive skills, combinations with more academically oriented interventions may yield complementary effects.

Prior reviews

In some regards, this review shares common ground with existing Campbell reviews and reviews in progress such as “Impacts of After-School Programs on Student Outcomes: A Systematic Review” (Zief et al. 2006), “Dropout Prevention and Intervention Programs: Effects on School Completion and Dropout among School-aged Children and Youth” (Wilson, Tanner-Smith, Lipsey, Steinka-Fry, & Morrison, 2011) and “Effects of College Access Programs on College Readiness and Enrollment” (Harvill et al., 2012).

Nevertheless, this review differs in substantial ways from existing Campbell reviews. First, with the exception of Zief et al. (2006), the listed reviews do not explicitly target an educationally disadvantaged or low performing student population. Zief et al. (2006) on the other hand excluded interventions performed outside North America, and three of the five studies included were of programmes primarily designed to reduce negative behaviours such as delinquency and drug use. That is, the programmes did not target academic achievement as their primary outcome. Although Wilson et al. (2011) did not explicitly include only interventions that targeted students with or at risk of academic difficulties, many of the studies in their review of dropout prevention programmes included similar at-risk groups as our review. The major difference between their review and ours is that they focused on programmes of school completion and dropout prevention, and outcome measures such as dropout and graduation rates. There is some overlap between the types of interventions included but also clear differences. For instance, none of the interventions included in our review involved components such as paid employment for students, community service programmes, or vocational training.

In addition to these Campbell reviews, there are other related reviews with a similar broad scope and a target group overlapping with ours to some degree. Slavin, Lake, and Groff (2009) reviewed programmes in middle and high school mathematics, whereas Slavin et al. (2008) reviewed reading programmes for middle and high schools. Fryer (2016) surveys randomised field experiments in all areas of education. However, these reviews included a broader range of programmes and student groups, not programmes targeting at-risk or low-performing students. Furthermore, Wanzek et al. (2006; including students in Grades 6–12) reviewed reading interventions directed at students in grades K-12 with learning disabilities, and Edmonds et al. (2009; Grades 6–12), Flynn et al. (2012; Grades 5–9), and Scammaca et al. (2015; Grades 4–12), Wanzek et al. (2013; Grades 4–12) and Baye et al. (2018; Grades 6–12) reviewed interventions for struggling readers. These reviews thus covered low achieving students, but not other at-risk students or areas other than reading. Gersten et al. (2009) examined four types of components of mathematics instruction for students with learning disabilities but did not include interventions for students at risk (or more general reasons for low performance than learning disabilities). Dietrichson et al. (2017) on the other hand included studies in both reading and mathematics and based inclusion on the proportion of students with low SES, but did not consider whether students had academic difficulties or not.

In terms of findings related to this review's primary outcome measures, the reviews that have focused on the effects of academic interventions on reading test scores all showed positive overall effect sizes, although there was considerable variation between the interventions in all of these reviews (Baye et al., 2018; Edmonds et al., 2009; Flynn et al., 2012; Scammaca et al., 2015; Slavin et al., 2008; Slavin et al., 2009; Wanzek et al., 2006, 2013). The six reviews of reading interventions directed to struggling readers reported positive effects in general but few reliable differences between effect sizes over reading domains (Edmonds et al., 2009; Flynn et al., 2012; Scammaca et al., 2015; Wanzek et al., 2006, 2013). An exception is that reading comprehension interventions were associated with significantly higher effect sizes than fluency interventions in Scammaca et al. (2015), but this difference disappeared when only standardised measures were considered. Baye et al. (2018) classified interventions by their main component and found the largest effects for tutoring by paid adults. Cooperative learning, whole-school approaches, and approaches focused on writing, content, strategy-focused, personalisation and group/personalisation rotation had positive effects of relatively similar magnitudes (we compare our results, including the magnitudes of effects, to those found in the earlier literature in section Agreements and disagreements with other studies or reviews).

Gersten et al. (2009) examined four components of mathematics instruction for students with learning disabilities, and found most support for approaches to instruction (e.g., explicit instruction, use of heuristics) and/or curriculum design, and providing formative assessment data and feedback to teachers. Dietrichson et al. (2017) examined interventions that have used standardised tests in reading and mathematics and categorised 14 intervention components mainly delimited by the instructional methods used. Tutoring, feedback and progress monitoring, and cooperative learning had the largest and most robust average effect sizes.

The best evidence syntheses by Slavin et al. (2008, 2009) both point to instructional-process programmes, especially programmes that incorporate cooperative learning, as having larger effects than curriculum-based interventions and CAI programmes. Slavin et al. (2009) found no indication that effect sizes differed between socioeconomically disadvantaged students and nondisadvantaged students. However, only a relatively small subset of studies reported results differentiated by SES, and the review did not contain information about whether the programmes that showed the largest effect sizes also had the largest effect sizes for disadvantaged students. Fryer (2016) found that high dosage tutoring and “managed professional development” programmes (e.g., Success for All, Reading Recovery) had the largest effect sizes of experiments conducted in schools.

Slavin et al. (2009) and Edmonds et al. (2009) reported that some programmes, which have been shown to be effective for younger students, may have smaller or no effects for older students. Effect sizes were smaller for older students also in Scammaca et al. (2015), although not significantly different. Fryer (2016) on the other hand concludes that “high dosage tutoring of adolescents seems to be as effective—if not more effective—than early childhood investments” (p. 78). Neither the question of whether interventions are less effective for older students, nor which components of interventions that are most important was settled in the reviews covered in this section.

The contribution of this review

Academic difficulties and lack of educational attainment are significant societal problems. Moreover, as shown by the Salamanca declaration from 1994 (UNESCO, 1994), there has been a great, and decades long, interest among policy makers in improving the inclusion of students with academic difficulties in mainstream schooling, and a desire to increase the number of empirically supported interventions for these student groups.

The main objective of this review is to provide policy makers and educational decision-makers at all levels—from governments to teachers—with evidence of the effectiveness of interventions aimed to improve the academic achievement of students with or at risk of academic difficulties in Grades 7–12. To this end, we compared the effects of interventions that differ in both instructional methods as well as the nature of the content taught.

We chose a broad scope in terms of the target group and type of intervention. We included interventions where the effects were measured by standardised tests in reading and mathematics because many interventions are not directed specifically to either subject and outcomes are therefore often measured in both (Dietrichson et al., 2017). Reading and mathematics are furthermore fundamental skills, which are important in more or less all school subjects and highly correlated with future educational and labour market success (OECD, 2016c). Earlier reviews of interventions aimed at similar target groups (e.g., Dietrichson et al., 2017; Gersten et al. 2009) have provided tentative evidence that similar types of interventions were effective for both struggling and low SES students, but stronger evidence of this is needed. In order to provide as complete a picture of the evidence as possible, the type of intervention and the target group were both kept deliberately broad. Including both students with and at risk of academic difficulties in the target group should decrease the risk of biasing the results due to omission of studies where information about either academic difficulties or at-risk status is available, but not both. Furthermore, making comparisons between intervention components within one review, rather than across reviews, should increase the likelihood of a fair comparison. It is easier within the scope of one review (rather than across reviews) to ensure that effect sizes are calculated in the same way, the definitions of intervention components are consistent, and that moderators are coded in the same way.

In isolation, this last argument suggests that all interventions aiming to improve educational achievement for our target population should be included. However, we also wanted to explore if certain interventions work better than others. The results in the reviews of Slavin et al. (2008, 2009) and Dietrichson et al. (2017), for example, point to substantial variation in effect sizes of test scores in reading and mathematics. Importantly, considerable variation was also found within types of interventions. For the exploration of variation in effect sizes, a broad scope review may be a disadvantage, as information about moderators that are important in order to explain variation for some types of interventions are not relevant for others. We have therefore limited the included interventions to those that are targeted, rather than universal, and performed in a regular school situation during the regular school year. This delimitation increases the probability that potentially important moderators are reported in a comparable way.

Earlier reviews with a comparable focus have either not included intervention components together with other moderators in a meta-regression, or only included broad categories of interventions. This risks confounding the effects of intervention components with for example participant characteristics. Furthermore, some reviews have coded interventions regarding the instructional methods used, or regarding the type of content taught, but not both (e.g., Dietrichson et al., 2017; Gersten et al., 2009; Scammaca et al., 2015). These analyses risk confounding methods of instruction with content.

Types of studies

According to our protocol, included studies should use a treatment-control group design or a comparison group design (Dietrichson, Bøg, Filges, & Klint Jørgensen, 2016). Included study designs were RCTs, including cluster-RCTs; QRCTs, where participants are allocated by means such as alternate allocation, person's birth date, the date of the week or month, case number or alphabetical order (we found no such designs though); and QES. To be included, QES had to credibly demonstrate that outcome differences between intervention and control groups is the effect of the intervention and not the result of systematic baseline differences between groups. That is, selection bias should not be driving the results. This assessment is included as a part of the risk of bias tool, which we elaborate on in the Risk of Bias section below. A fair amount of studies within educational research use single group pre-post comparisons (e.g., Edmonds et al., 2009; Wanzek et al., 2006); such studies were however not included due to the higher risk of bias.

Control groups received treatment-as-usual (TAU) or a placebo treatment. We found no studies where the control group explicitly received nothing (i.e., a no-treatment control), as everybody experienced regular schooling. That is, control groups got whatever instruction the intervention group would have gotten, had there not been an intervention. The TAU condition can for this reason differ substantially between studies (although many studies did not describe the control condition in much detail). Eligible types of control groups included also waiting list control groups, in which the control group also receives the intervention after the posttest.

Comparison designs compared alternative interventions against each other; that is, they made it clear that all students get something other than TAU because of the intervention. Effect sizes from such studies are not fully comparable to effect sizes from treatment-control designs. We therefore planned to analyse comparison designs separately from treatment-control designs, and use them where they may shed light on an issue, which could not be fully analysed using the sample of treatment-control studies. However, the number of studies that were, in this sense, relevant was small and we refrained from analysing comparison designs further.

Types of participants

The population samples eligible for the review included students attending regular schools in Grades 7–12, who were having academic difficulties, or were at risk of such difficulties.

Students attending regular private, public, and boarding schools were included, and students receiving special education services within these school settings were also included. Grades 7–12 corresponds roughly to secondary school, defined as the second step in a three-tier educational system consisting of primary education, secondary education and tertiary or higher education. We included studies with a student population in grades lower than 7–12 as long as the majority of the students were in Grades 7–12. The age range included differed between countries, and sometimes between states within countries. Typically, ages ranged from 12–14 to 17–19 years (fewer studies reported ages than grades though, which was also the main reason to focus inclusion on grades rather than age).

The eligible student population included both students identified in the studies by their observed academic achievement (e.g., low academic test results, low grade point average or students with specific academic difficulties such as learning disabilities), and students that were identified primarily on the basis of their educational, psychological or social background (e.g., students from families with low SES, students placed in care, students from minority ethnic/cultural backgrounds and second language learners). We excluded interventions only targeting students with physical learning disabilities (e.g., blind students), students with dyslexia/dyscalculia, and interventions that were specifically directed towards students with a certain neuropsychiatric disorder (e.g., autism, ADHD), as some interventions targeting such students are different from interventions targeting the general struggling or at-risk student population.

Because there was substantial overlap between students that were already struggling and groups considered at risk of difficulties in studies found in a previous review (Dietrichson et al. 2017), we chose to include both students with difficulties and students that were deemed at risk, or were considered educationally disadvantaged. A motivating example is studies that target a high poverty area, and then randomly select a number of students with test scores below a certain level in each school that receive the intervention. These struggling students are thus likely to be low SES, but information about SES is not necessarily reported for the intervention or control group. A second example could be studies that target low performing schools, and then perform an intervention for the sub-group of low SES students. In this case, low SES students are likely to be struggling, but this information need not be reported. Thus, choosing to include only studies that examine either students with academic difficulties or low SES students would have risked excluding studies that in all likelihood target the same or very similar student populations. We believed that the risk of biasing our results by such a choice would be larger than the possible comparison problems arising from including both students with academic difficulties and low SES students. A similar case can be made for other at risk groups, for example students from immigrant and minority backgrounds, which often partly overlap with low SES students. If the two criteria were inconsistent, we gave priority to students having academic difficulties. For example, we excluded interventions that targeted high achieving students from low income backgrounds.

Some interventions included other students, who had neither academic difficulties nor were at risk of such difficulties. For example, in some peer-assisted learning interventions high performing students were paired with struggling students. Studies of such interventions were included if the total sample (intervention and control group) included at least 50% students that were either having academic difficulties or were at risk of developing such difficulties, or if there were separate effect sizes reported for these groups.

Types of interventions

We included interventions that sought to improve academic achievement or specific academic skills. This does not mean that the intervention had to consist of academic activities, but there had to be an explicit expectation in the study that the intervention, regardless of the nature of the intervention content, would result in improved academic achievement or a higher skill level in a specific academic task. We however choose to exclude interventions that only sought to improve performance on a single test instead of improving a skill that would improve test scores. For similar reasons, we excluded studies of interventions where students are provided with accommodations when taking tests; for instance, when some students are allowed to use calculators and others not.

An explicit academic aim of the intervention did not per se exclude interventions that also included nonacademic objectives and outcomes. However, we excluded interventions having academic learning as a possible, not explicitly stated, secondary goal. In cases where the goals were not explicitly stated, we used the presence of a standardised test in mathematics or reading as a sign that the authors expected the intervention to improve academic achievement. We excluded cases where such tests were included but the authors explicitly stated that they did not expect the intervention to improve reading or math. Furthermore, we only included school-based interventions; that is, interventions performed in schools during the regular school year, and where schools were one of the stakeholders. This latter restriction excluded summer reading programmes, after-school programmes, parent tutoring programmes and other programmes delivered in the home of students.

Besides having an explicit expectation that the intervention would improve the academic performance of students, eligible interventions were also targeted (selected or indicated). That is, interventions which, in contrast to universal interventions, were aimed at certain students and/or student groups identified as having or being at risk of academic difficulties according to the definition in the previous section. Universal interventions that aimed to improve the quality of the common learning environment at the school level in order to raise academic achievement of all students (including average and above average students), were excluded. Interventions such as the one described in Fryer (2014) where a bundle of best practices were implemented at the school level in low achieving schools, where most students are struggling or at risk, was also excluded. This criterion also excluded whole-school reform strategy concepts such as Success for All, curriculum-based programmes like EMP, as well as reduced class size interventions.

This criterion also meant that we excluded interventions where teachers or principals receive professional development training in order to improve general teaching or management skills. Interventions targeting students with or at risk of academic difficulties may on the other hand include a professional development component, for example when a reading programme includes providing teachers with reading coaches. Such interventions were therefore included.

Types of outcome measures

We included outcomes that cover two areas of fundamental academic skills:

Standardised tests in reading

Studies were only included if they considered one or more of the primary outcomes. Standardised tests included norm-referenced tests (e.g., Gates-MacGinitie Reading Tests and Star Math), state-wide tests (e.g., Iowa Test of Basic Skills) and national tests (e.g., National Assessment of Educational Progress [NAEP]). If it was not clear from the description of the outcome measures in the studies, we used electronic sources to determine whether a test was standardised or not. For example, if a commercial test has been normed, this was typically mentioned on the publisher's homepage. For older tests however it was not always possible to find information about the test from electronic sources. In these cases, we included the test if there was a reference to a publication describing the test that made it clear that the test had not been developed for the intervention or study.

We restricted our attention to standardised tests in part to increase the comparability between effect sizes. Earlier related reviews of academic interventions have pointed out that effect sizes tend to be significantly lower for standardised tests compared to researcher-developed tests (e.g., Flynn et al., 2012; Gersten et al., 2009; Scammaca et al., 2015). Scammaca et al. (2015) furthermore reported that whereas mean effect sizes differed significantly between the periods 1980-2004 and 2005-2011 for other types of tests, mean effect sizes were not significantly different for standardised tests. As researcher developed tests are usually less comprehensive and more likely to measure aspects of content inherent to intervention but not control group instruction (Slavin & Madden, 2011), standardised tests should provide a more reliable measure of lasting differences between intervention and control groups.

We excluded tests that provided composite results for several academic subjects, but included tests of specific domains (e.g., vocabulary, fractions) as well as more general tests, which tested several domains of reading or mathematics. Tests of subdomains had significantly larger effect sizes compared to more general tests in Dietrichson et al. (2017). This result may indicate that it may be easier to improve scores on tests of subdomains than on tests of more general skills, or that tests of subdomains may be more likely to be inherent to intervention group instruction. At the same time, it seems reasonable that interventions that target subdomains of reading and mathematics are tested on whether they affect these subdomains. Therefore, we did not want to exclude either type of test, but coded the type of test and used it as a moderator in the analysis. However, to mitigate problems with test content being inherent to intervention and not control group instruction, we did not consider tests where researchers themselves picked a subset of questions from a norm-referenced test as being standardised. The subset should either have been predefined (as in e.g., the passage comprehension subset of Woodcock-Johnson Tests of Achievement) or the picked by someone other than the researchers (e.g., released items from the NAEP).

We included all postintervention tests and coded the timing of each test (see the Multiple Time Points section below).

Duration of follow-up

Our protocol contained no initial criterion for the duration of interventions, we included interventions of all durations. Duration of the intervention was included as a moderator in some of the analyses.

Types of settings

Only studies carried out in OECD countries were included. This selection was conducted to ensure a certain degree of comparability between school settings and to align treatment as usual conditions in included studies. For similar reasons we only included studies published in or after 1980. Due to language restrictions in the review team, we included studies written in English, German, Danish, Norwegian and Swedish.

Electronic searches

Relevant studies were identified through electronic searches of bibliographic databases, government and policy databanks. The following electronic resources/databases were searched:

Academic Search Premier (EBSCO-host)

All databases were originally searched from 1st of January 1980 to 5th–7th of March 2016. We updated the searches in July 2018 using identical search strings. However, some database searches were not updated in 2018 due to access limitations. Details on searches are listed in Appendix Search Strategy by Database.

The search terms were modified to fit each resource searched. In Appendix Search Strategy by Database, we report an example of the original search string for each host/search platform (ERIC for EBSCO, Social Science Citation Index for ISI Web of Science, Medline for OVID, and ProQuest dissertation & theses A&I for ProQuest). We used the same search string, with minor modifications, for each platform.

Note that the searches contained terms relating to primary school, since the search contributed to a review about this younger age group (kindergarten to Grade 6, see Dietrichson, Bøg, Eiberg, Filges, & Klint Jørgensen, 2016, for the protocol). There is overlap in the literature among the age groups, and in order to rationalise and accelerate the screening process, we decided upon performing one extensive search.

Searches on national indices/repositories

Australian Education Index (ProQuest-host)

All indices and repositories were originally searched from 1st of January 1980 to 2nd–11th of March 2016. We updated the searches in July 2018 using identical search strings.

Contact to international experts

We contacted international experts to identify unpublished and ongoing studies. We primarily contacted corresponding authors of the related reviews mentioned in Section 3.4.1,

Citation tracking

In order to identify both published studies and grey literature we utilised citation-tracking/snowballing strategies. Our primary strategy was to citation-track related systematic-reviews and meta-analyses: 1,446 references from 23 existing reviews were screened in order to find further relevant grey and published studies (see Section 3.4.1 and the list in the appendix Grey Literature and Searches on Other Resources). The review team also checked reference lists of included primary studies for new leads.

Trial registries

Our protocol stated that we should search two trial registries: The Institute for Education Sciences’ (IES) Registry of Randomized Controlled Trials (http://ies.ed.gov/ncee/wwc/references/registries/index.aspx), and American Economic Association's RCT Registry (https://www.socialscienceregistry.org). We were however unable to search the IES registry as it was not available (last tried in July 23, 2018). We have asked IES about availability, but have to date not received a reply. We updated the search of American Economic Association's RCT Registry in July 23, 2018.

Hand searches

The following international journals were hand searched for relevant studies:

American Educational Research Journal

The search was performed on editions from 2015 to July 2018 (i.e., including an updated search) of the journals mentioned, in order to capture relevant studies recently published and therefore not found in the systematic search.

Grey literature

Different strategies were utilised in order to identify relevant grey literature. A wide range of searches were performed on the below institutional and governmental sites, academic clearinghouses and repositories for relevant academic theses, reports and conference/working papers:

What Works Clearinghouse—U.S. Department of Education (whatworks.ed.gov)

Selection of studies

Under the supervision of the review authors, at least two review team assistants independently screened titles and abstracts to exclude studies that were clearly irrelevant. Any disagreement of eligibility was resolved by the review authors. Studies considered eligible were retrieved in full text. The full texts were then screened independently by two review team assistants under the supervision of the review authors. Any disagreement of eligibility was resolved by the review authors. The study inclusion criteria were piloted by the review authors with all members of the review team.

Data extraction and management

Two members of the review team independently coded and extracted data from included studies. A coding sheet was piloted on several studies and revised. Any disagreements were resolved by discussion, and it was possible to reach consensus in all cases. Data was extracted on the characteristics of participants, characteristics of the intervention and control/comparison conditions, research design, sample size, outcomes and results. Extracted data was stored electronically, and we used EPPI Reviewer 4, Microsoft Excel, R and Stata as the primary software tools.

Assessment of risk of bias in included studies

We assessed the risk of bias of effect estimates using a model developed by Prof. Barnaby Reeves in association with the Cochrane Non-Randomised Studies Methods Group. This model is an extension of the Cochrane Collaboration's risk of bias tool and covers risk of bias in nonrandomised studies that have a well-defined control group. The extended model is organised and follows the same steps as the risk of bias model according to the 2008-version of the Cochrane Hand book, chapter 8 (Higgins & Green, 2008). The extension to the model is explained in the three following points: (1)

The extended model specifically incorporates a formalised and structured approach for the assessment of selection bias in nonrandomised studies by adding an explicit item about confounding. This is based on a list of confounders considered to be important and defined in the protocol for the review. The assessment of confounding is made using a worksheet where, for each confounder, it is marked whether the confounder was considered by the researchers, the precision with which it was measured, the imbalance between groups, and the care with which adjustment was carried out. This assessment informed the final risk of bias score for confounding.

The refined assessment is pertinent when thinking of data synthesis as it operationalizes the identification of studies (especially in relation to nonrandomised studies) with a very high risk of bias. The refinement increases transparency in assessment judgements and provides justification for not including a study with a very high risk of bias in the meta-analysis.

Measures of the intervention effect

The analysis of effect sizes involved comparing an intervention to a control condition. We conducted separate analyses for short- and long(er)-term outcomes, although this latter analysis was highly constrained due to the lack of long-term outcomes in the included studies. The analysis plan laid out below applies to both types of outcomes.

Effect sizes using continuous data

For continuous data, standardised mean differences (SMDs) were calculated when means and standard deviations were available. All studies included in the data synthesis provided information so that student level effect sizes could be calculated. We used Hedges' g to estimate SMDs where scales have been used to measure the same outcomes in different ways. Hedges' g and its standard error were calculated as (Lipsey & Wilson, 2001, pp. 47–49):

g = ( 1 − 3 4 N − 9 ) × ( X ¯ 1 − X ¯ 2 s p ) ,

1

S E g = N n 1 n 2 + g 2 2 N ,

2

where

N = n 1 + n 2

is the total sample size,

X ̅

is a postintervention mean in each group and

s p

is the pooled standard deviation defined as

s p = ( n 1 − 1 ) s 1 2 + ( n 2 − 1 ) s 2 2 ( n 1 − 1 ) + ( n 2 − 1 ) ,

3

here,

s 1

and

s 2

denotes the standard deviation of the intervention and control group. We used covariate adjusted means, and the unadjusted posttest standard deviation whenever available. We decided to use the postintervention standard deviation, as more studies included this information than the preintervention standard deviation. In the few cases where instead the postintervention standard deviation was missing, we used the preintervention standard deviation.

Some studies reported only raw means. In these cases, we coded two effect sizes: one based on the unadjusted postintervention mean difference (i.e., exactly as in Equation (1) above), and one where we subtracted the preintervention mean from the postintervention mean for both the intervention and control group. That is,

g = ( 1 − 3 4 N − 9 ) × ( ( X ¯ 12 − X ¯ 11 ) − ( X ¯ 22 − X ¯ 21 ) s p ) ,

4

where

X ̅ 12

is the postintervention mean in the intervention group,

X ̅ 11

is the preintervention mean in the intervention group, and

X ̅ 22

and

X ̅ 21

are the corresponding means in the control group. We used the effect sizes based on Equation (1), which was available in more studies, as our baseline effect size and used the second type in a sensitivity test (reported in the Results of Sensitivity Analyses section).

Six studies reported an effect size where the mean difference was standardised using the control group's standard deviation (i.e., Glass's delta), and two studies used, for example, the school-, district-, or nation-wide standard deviation, but did not include information about the respective standard deviation for intervention and control group. We included these effect sizes and tested the sensitivity to their inclusion in the Results of Sensitivity Analyses section.

Our protocol stated that we would use intention-to-treat (ITT) estimates of the mean difference whenever possible. However, very few studies reported explicit ITTs, and the overwhelming majority only reported results for the students that actually received the intervention, rather than all for which the intervention was intended (often because they lacked outcome data for students that left the study). We therefore believe that the estimates are closer to treatment-on-the-treated (TOT) effects and used TOT estimates in the few cases where both ITTs and TOTs were available. We tested the sensitivity to this choice in the Results of Sensitivity Analyses section.

Unit of analysis issues

Errors in statistical analysis can occur when the unit of allocation differs from the unit of analysis. In cluster randomised trials, participants are randomised to intervention and control groups in clusters, as when participants are randomised by treatment locality or school. QES may also include clustered assignment of treatment. Effect sizes and standard errors from such studies may be biased if the unit-of-analysis is the individual and an appropriate cluster adjustment is not used (Higgins & Green, 2011).

If the information was available, we adjusted the effect sizes using the methods suggested by Hedges (2007) and information about the intracluster correlation coefficient (ICC), realised cluster sizes, and/or estimates of the within and between variances of clusters. If this information was not available, we instead used estimates from the literature of the ICC and assumed equal cluster sizes. We calculated our ICCs by taking the average over the ICCs reported for low achievement schools in Grades 7–12 in column 1 of tables 6 (mathematics) or 7 (reading) in Hedges and Hedberg (2007). To calculate an average cluster size, we divided the total sample size in a study by the number of clusters (typically the number of classrooms or schools).

Most studies lacked information about one of more of the ICC, cluster sizes, and within and between variances. As the procedure therefore entailed considerable uncertainty about how correct the adjustment was, we used the effect sizes adjusted with estimates from the previous literature only in the sensitivity analysis. Effect sizes that could be individually adjusted using information from the study (or directly provided adjusted effect sizes) were adjusted (used) also in the main analysis.

Dealing with missing data

Missing data and attrition rates in the individual studies were assessed using the risk of bias tool. Studies had to permit a calculation of a numeric effect size for the outcomes to be eligible for inclusion in the meta-analysis. Where studies had missing summary data, such as missing standard deviations, we derived effect sizes where possible from, for example, F ratios, t values, χ ² values and correlation coefficients using the methods suggested by Lipsey and Wilson (2001). If these statistics were also missing, we asked the study investigators if they could provide us with the information. We were unable to retrieve information from 13 studies. These studies were included in the review but excluded from the meta-analysis (see Appendix Studies with Overlapping Samples or Lacking Information, Table A3, for a summary description).

Many studies did not provide data about all moderators. We focused on moderators that were relevant for all types of studies and used multiple imputation methods (see, e.g., Rubin, 1996, and Pigott, 2009) in a sensitivity analysis to test if our results were sensitive to omitting moderators. We used the Stata command mi impute with sequential imputation using chained equations to generate values for missing observations. All variables without missing observations were used in the estimation to impute values for variables with missing observations. We used 20 imputed data sets.

Assessment of heterogeneity

Heterogeneity was assessed with χ ² (Q) test, and the I ², and τ ² statistics (Higgins, Thompson, Deeks, & Altman, 2003).

Assessment of reporting biases

Reporting bias refers to both publication bias and selective reporting of outcome data and results. Bias from selective reporting of outcome data and results is one of the main items in the risk of bias tool.

To examine possible publication bias, we used funnel plots, Egger's test, and tested whether studies published in scientific journals had different effect sizes compared to other studies.

Data synthesis

The overall data synthesis in this review was conducted when effect sizes were available. Effect sizes coded with a very high risk of bias (score of 5 on any item judged on a 5-point scale) were not included in the data synthesis. The analysis was conducted in the following steps. We described summary and descriptive statistics of the intervention-level characteristics, and the risk of bias assessment. We also included a correlation matrix with all moderators. We performed analyses divided by measurement timing first (end-of-intervention or follow-up). To be able to simultaneously include all effect sizes from each study and avoid problems with dependence between effect sizes we used the RVE procedure in the Stata command robumeta for estimation and to calculate robust standard errors (Hedges et al., 2010). We used the random effects model weighting scheme option, as it seemed most likely that the effects of the included interventions were not the same across studies, but follow a distribution. A fixed effect model would therefore be less appropriate in our case (e.g., Borenstein, Hedges, Higgins, & Rothstein, 2009).

The RVE procedure requires an initial estimate,

ρ ˆ

ρ ˆ

Subgroup analysis and investigation of heterogeneity

The analysis revealed substantial heterogeneity in effect sizes and one of the main objectives in the review was to assess the comparative effectiveness of intervention components. We therefore performed subgroup and moderator analysis to attempt to identify the characteristics of study methods, interventions, and participant characteristics that were associated with smaller and larger effects. We used meta-regressions to reduce the risk of misleading results due to correlated independent variables, and again used the RVE procedure in robumeta. We reported 95% confidence intervals for regression coefficients.

Most included moderators were coded as indicator variables (most variables are natural indicators, e.g., whether the study design was an RCT or not). Continuous variables were mean-centred to facilitate interpretation. Our protocol specified the following types of moderators:

Subject

Below we describe the moderators we used. However, the number of included studies and effect sizes were not large enough to include all coded moderators in one meta-regression (see the Appendix and the Coding Scheme Section for a description of all coded variables). In line with the objective of the review and our protocol, we therefore focused the analysis of subgroups and heterogeneity on instructional methods and content domains. These are substantive features of interventions that for example teachers and principals can affect, in contrast to other moderators (e.g., participant characteristics may be more difficult to affect for a school). They were also more often (quasi-)experimentally manipulated in studies than other moderators in our sample. They may therefore be less likely to be confounded with other, omitted moderators.

To further reduce the number of moderators, we first excluded moderators with very low variation (i.e., where nearly all observations have the same value) or where information was missing from studies. We also excluded moderators that were not relevant for all intervention types (for example, there is no number of sessions in an intervention that provide students with incentives to read a certain number of books). For some analyses we excluded highly correlated variables.

We have characterised the included interventions by their components using two general categories of treatment modalities: instructional method and content domain. As described in our protocol, the components were not fully prespecified, but developed and adapted during the coding process. We used previous reviews and author-reported classifications in included studies as a starting point, and an iterative process to construct component categories. Below, we describe the coded components by treatment modality, and how these components were used to develop the moderators we included in the meta-regressions. Note that interventions often contained more than one component and they were coded in all component categories they contained. The categories below are therefore not mutually exclusive.

Sensitivity analysis

We performed the following sensitivity analyses:

Results of the search

Figure 1 displays the results of the search process. The total number of potentially relevant records was 24,411 after excluding 187 duplicates (database search: 17,444; grey literature: 3,014; citation tracking: 1,508; author contacts: 576; hand search: 1,024; trial registries and others: 845).

OPEN IN VIEWER

Figure 1

Flowchart of the search and screening process

All records were screened based on title and abstract. Of the ones that were not excluded, 201 records were not retrievable in full text. Older reports and dissertations were overrepresented among these records. The remaining 4,244 retrievable records were screened in full text. A large number of studies were not relevant for this review due to the grade of the participating students. Studies that were relevant except for the grade of participating students will be included in a future review covering kindergarten to Grade 6 (see Dietrichson, Bøg, Eiberg, et al., 2016 for the protocol). Two hundred forty seven studies met the inclusion criteria for this review and data were extracted from these studies.

Included studies

The number of included studies was 247. Of these studies, we included 71 in meta-analyses. We were unable to retrieve sufficient information from 13 studies/study authors to calculate an effect size. Seven studies used samples that overlapped with other included studies and had either a higher risk of bias or contained less information. These studies were not included in the analysis. Some studies contained overlapping samples but included, for example, information about short- and long-run outcomes. These were all included in some meta-analysis, but never in the same. Thirty-eight studies were not included in the meta-analyses due to their study design. These studies used comparison designs that contrasted two alternative interventions. We did not include 118 studies in the meta-analysis due to the risk of bias assessment. All eligible outcomes from these studies were excluded due to too high risk of bias.

We discuss the results of the risk of bias assessment further in the Risk of Bias in Included Studies section. See also the Risk of Bias Tables in the online appendix for details of the assessment for effect sizes that we included in the meta-analysis, as well as those that we deemed had too high risk of bias. The comparison designs are described in the appendix, in the Comparison Designs section, Table A2. For more information about studies with overlapping samples or that lacked sufficient information for the calculation of an effect size, see the online appendix, section Studies with Overlapping Samples or Lacking Information, Table A3.

The 71 studies included in the meta-analyses are described in detail in the Studies Included in the Meta-Analysis section, Table A1, in the appendix. Figure 2 displays the 71 studies by publication year. Studies with more than one intervention or where different cohorts received the same intervention were treated as one study. There is an increasing trend, only six included studies were published before the year 2000 and about half the studies have been published since 2010. Figure 3 shows the 71 studies by the mean grade of participating students. The bulk of studies included participants in the lower grades (e.g., the mean grade is around seven in 30 studies). There were few studies of interventions in Grades 11 and 12.

OPEN IN VIEWER

Figure 2

Number of studies included in the meta-analysis by year

OPEN IN VIEWER

Figure 3

Number of studies included in the meta-analysis by mean grade

Table 1 contains descriptive statistics of the studies included in the meta-analysis. Many studies contained more than one intervention, the effects of which may have been tested with more than one standardised test from which we calculated the effect sizes. We denoted the number of studies that provided information about a certain characteristic with k, the number of interventions with i,

Included interventions were to a large extent, 87%, performed in the United States. The remaining interventions were from Canada (4%), Germany (4%), the United Kingdom (UK; 3%), the Netherlands (2%) and Australia (1%).

There were more interventions using reading tests, 32% used a mathematics test. Note that the separation of effect sizes was made based on the test, not the subject targeted, as there were several interventions (i = 11) that used tests in both reading and mathematics. This is also the main reason why we do not separate results into reading and mathematics interventions to start with (we will return to this issue in the analysis of heterogeneity).

The mean duration was about 27 weeks, and the mean frequency and intensity equalled 75 sessions and 3.0 hr per week. The ranges for all three variables measuring dosage were wide and note that we lack information for quite a few interventions regarding frequency and intensity (for 22 and 29 interventions, respectively). The same is true to a somewhat lesser extent for the variables characterising the implementer. Among the interventions with information, 68% was implemented by school staff, and 20% by researchers, and the rest by other categories, such as college students.

The proportion of instructional methods ranged from 12% of interventions using an incentive component to 33% that used small group instruction. In 14% of the interventions (i = 14) there was no component included that matched the categories laid out in the Instructional Methods section, and they were coded in the other method-category. Eight of these interventions contained instruction in groups smaller than whole class, but larger than five students. Three provided extra instructional time and three changed only the content, not the instructional method.

Nearly half of the interventions (46%) combined more than one instructional method. Accordingly, as shown at the bottom of the table, the share of interventions using a single instructional method was, for most methods, small. The exceptions are peer-assisted instruction, which was the only instructional method-component in 8% of the interventions, and small group instruction, which was the only instructional method-component in 21% of the interventions.

The proportion of interventions targeting reading domains ranged from 17% that targeted fluency to 45% that targeted comprehension. Most interventions (51%) targeted multiple reading areas. For most math domains there were few interventions and we were unable to analyse heterogeneity over math domains any further. Lastly, while there were many interventions targeting meta-cognitive strategies (43%), fewer targeted social-emotional skills (8%) or general academic skills (12%). In the rest of the analysis, we combined meta-cognitive strategies, social-emotional skills, and general academic skills into one domain, called other domain, due to the low number of interventions targeting the latter two domains. None of the domains was the only targeted domain in more than 10% of the interventions.

Excluded studies

Due to the large amount of studies screened in full text, we were unable to describe all excluded studies. This section instead describes studies that met almost all our inclusion criteria and that one may have thought should be included. They therefore exemplify how we applied the inclusion criteria.

The included study designs contrasted intervention and control groups, or alternative interventions, to estimate effects. Gajria and Salvia (1992) used a randomised design where one intervention group consisting of students with learning disabilities were given training in summarisation strategies. The control group of students with learning disabilities did not receive this training. Results were also compared to a group of students without learning disabilities, which we deemed to be not comparable to the intervention group. Furthermore, only the intervention group was given a standardised postintervention test (there were researcher developed tests given to all groups). For the measure we wanted to include, the design was therefore a single group pre-post design, and we excluded the study.

We included interventions that sought to improve academic achievement or specific academic skills. We however excluded interventions that only sought to improve performance on a single test instead of improving a skill that would improve test scores. Two studies about incentives provide contrasting examples: Levitt, List, Neckermann and Sadoff (2016) provided students with incentives to perform well on a single test, where students were unaware of the incentives right up until they arrived at the testing session. In Levitt, List, and Sadoff (2016) students were provided monthly financial incentives for meeting achievement standards based on multiple measures of performance including attendance, behaviour, grades and standardised test scores. We excluded the former study and included the latter, as the former did not target academic skill building, just motivation to perform on a particular test, whereas the latter did target skill building.

Interventions had to be targeting students with or at risk of academic difficulties to be included. Glazerman et al. (2013) examined a transfer-incentive programme for high-performing teachers, which aimed to increase the supply of effective teachers in struggling schools. Although these schools served disadvantaged students, we excluded the study because the programme did not specifically target students with or at risk of academic difficulties, but all students, including high-performing students, in these schools. Voight and Velez (2018) study an intervention where more than 50% of the participants were minority students and students receiving free- or reduced price lunches. However, as the intervention group average was higher than the national average on a normed test, the intervention included mainly high-performing students. Therefore, we excluded the study.

Interventions should be school-based to be included, meaning that they were performed in school during regular school-hours and semesters. Munoz et al. (2008) and Munoz et al. (2012) studied tutoring-programmes for at risk students within the United States Supplemental Educational Services programme. As the tutoring was only offered outside of regular school hours (e.g., before or after school, on weekends, or during the summer) by other providers than schools themselves, we excluded these studies because the interventions were not school-based.

Studies had to test the effects of interventions using standardised tests in reading and mathematics. Reyes and Jason (1991) used the Test of Academic Proficiency, which is a standardised test with a composite score in reading, mathematics, and writing. As we did not include writing tests, and it is unclear whether composite scores can be compared to tests in reading or mathematics, we excluded the study.

Overall short-run and follow-up effects

This section presents the results from the robust variance estimation of short-run effects and effects at follow-up. Note that the number of clusters used in the robust-variance estimation differs from the total number of included studies. Somers et al. (2010) was excluded in the analyses of short-run effects. This study included a mix of the samples in Kemple et al. (2008) and Corrin et al. (2008), which examined the same intervention and were consequently treated as one cluster. Results from Somers et al. (2010) are included in the follow-up analysis, as none of the other two studies provided results from follow-up tests. Kim et al. (2011) and Olson et al. (2012) report estimates from the year 1 cohort and year 2 cohort of the same intervention. Vaughn et al. (2011) and Swanson et al. (2015) similarly report effects for two cohorts of one intervention. The effect sizes in these two pairs of studies are therefore unlikely to be independent. We included all effect sizes from these four studies but clustered on the intervention level rather than the study level.

There were 194 short-run effect sizes, 66 clusters, and 79,191 student observations.

OPEN IN VIEWER

Figure 5

Study level weighted effect sizes with outcomes measured at the end of interventions

There were 16 effect sizes from seven clusters (equal to the number of studies in this case) measured more than 3 months after the end of intervention, including 21,630 student observations. The ES at follow-up was small and not statistically significant (ES = 0.05, CI = [−0.096, 0.192]). The number of studies and the degrees of freedom are low enough that the RVE procedure may yield untrustworthy results regarding the standard errors (Tanner-Smith & Tipton, 2014; Tipton, 2015). However, we got similar results when we, instead of using RVE, averaged effect sizes over studies and used the DerSimonian-Laird (DL) procedure in the Stata command metan (ES = 0.03, CI = [−0.051, 0.117]).

As there were few follow-up studies and effects, the subgroup analysis and investigation of heterogeneity in the following sections focus only on the short-run effects.

Results of the subgroup analysis and investigation of heterogeneity

The previous analyses all indicated substantial heterogeneity. This section examines if we can explain some of this heterogeneity by dividing studies further into subgroups by instructional components and content domains, and by using meta-regressions to examine the associations between intervention characteristics and effect sizes.

Figures 6 and 7 show ESs and 95%-confidence intervals from RVE estimations by instructional method (Figure 6) and content domains (Figure 7). We derived each effect size from a meta-regression including just a constant with the outcome variable being the effect sizes from interventions including the component in question. Note that many interventions have more than one component, so the effect sizes should be interpreted as the ES for interventions that included a certain component, not the effect size of that component in isolation. Furthermore, many interventions included several components and contributed effect sizes to more than component category. Because there were few studies and interventions that studied subdomains of mathematics (see Table 1), we used a single math-domain. The effect sizes in the math-domain were from standardised mathematics tests.

OPEN IN VIEWER

Figure 6

Subgroup analyses: Weighted average effect sizes (ESs) and 95%-confidence intervals by instructional component

OPEN IN VIEWER

Figure 7

Subgroup analyses: Weighted average effect sizes (ESs) and 95%-confidence intervals by content domain

Figure 6 indicates that all instructional components were associated with positive average effect sizes. Small group instruction is associated with the largest effect sizes (ES = 0.38, CI = [0.211, 0.547]) and incentives with the smallest (ES = 0.046, CI = [−0.103, 0.194]). The average effect size is also significantly different from zero for all instructional components, except for incentives and the other method category. There are three main messages from Figure 7: First, all content domains have positive and statistically significant average effect sizes. Second, the differences between the reading domains were small. The average effect sizes were around 0.14-0.22 and none of them were significantly different from each other. Third, the average effect size for math (ES = 0.34, CI = [0.169, 0.502]) was considerably larger than most other domains.

Table 2 summarises the results for both types of components and includes the number of studies and effect sizes per component as well as results for the three heterogeneity measures. For almost all components, the heterogeneity measures indicated substantial heterogeneity. Incentives and coaching of personnel were partial exceptions, but the Q statistic was significant also for these components.

This component-by-component analysis has a few important drawbacks. If there is correlation between components, that is, when interventions contain more than one component, we risk confounding the association of one component with the others. As shown in Table 1, several components were always used in combination with at least one other component and there are many combinations, so this risk is pertinent. One way to mitigate this problem would be to examine interventions that only included one instructional method or one content domain and run separate regressions using only effect sizes from these interventions.

However, for most components, we encountered degrees of freedom problems in the RVE procedure when we examined single component interventions. The exceptions for instructional methods were peer-assisted instruction and small group instruction, which both showed large, positive, and significant associations with effect sizes (ES = 0.44, CI = [0.011, 0.869] for peer-assisted instruction and ES = 0.56, CI = [0.307, 0.808] for small group instruction). Regarding content domains, it was only the combined category for meta-cognitive, social-emotional, and general skills (i.e., the other domain) and the combined categories for math and reading that did not have degrees of freedom problems. The average effect size was positive but not statistically significant for the other domain (ES = 0.20, CI = [−0.119, 0.519]), and positive and significant for math (ES = 0.35, CI = [0.047, 0.656]) and reading (ES = 0.20, CI = [0.109, 0.293]).

The component-by-component analyses do not take into account other moderators and we therefore risk confounding the instructional components and content domains with other study, participant and intervention characteristics. Next, we therefore performed meta-regressions where we included indicators of intervention components and moderators based on study, participant and intervention characteristics in the same regression. However, as discussed in the Subgroup Analysis and Investigation of Heterogeneity section, there were not enough studies and effect sizes to include all potentially important moderators. To prune the set of moderators, we included only moderators without missing observations and then examined the correlations between them. Table 3 shows a matrix with the pairwise correlation between effect sizes (the two continuous variables, grade and duration, were mean-centred). Correlations over 0.5 are marked in bold. Note in particular that moderators indicating content domains were highly correlated with each other. There were fewer highly correlated moderators among the instructional methods, but the correlations between CAI and incentives, and CAI and QES were higher than 0.5. There were also relatively high negative correlations between the indicator for mathematics test and some of the reading domains.

High correlations between moderators increases the risk of multicollinearity and makes it less likely that we can estimate meaningful separate associations. Considering the correlations in Table 3, this problem seems to be particularly high for content domains. The correlations between instructional methods in Table 3 are not as high as for content domains (but they are clearly not zero). We therefore focused the further analysis on the instructional components.

Table 4 displays the results from meta-regressions where we included all end of intervention effect sizes from both math and reading tests. The component estimates are estimated from both within- and between-study variation, as some studies included more than one intervention, which in turn included different components. However, there is much more between-study variation.

In column 1, we included a single moderator indicating if the basis for the effect size was a mathematics test. The coefficient of the constant can therefore be interpreted as the average effect size based on reading tests. Column 2 includes a single indicator for QES, so the coefficient of the constant represents the average effect size in RCTs. The specification in column 3 includes both indicators for QES and math-tests; the constant is therefore the average effect size from reading tests in RCTs. In column 4, we added the six instructional components and a moderator indicating that the intervention targeted other domains than (or possibly in addition to) reading or math content. The constant represents the average effect size from reading tests in RCTs of interventions that did not use any of the included components (i.e., interventions coded in the other method category that did not focus on meta-cognitive, social-emotional or general academic skills). Finally, column 5 includes indicators for general tests and implementation problems, the mean grade, and the mean duration along with all the previous moderators. The constant becomes harder to interpret in this regression. It represents the average effect size from reading tests in RCTs of interventions that did not use any of the included components, and had a value of zero on the three added variables (i.e., was exactly at the mean). However, no effect size is likely to be exactly at the mean value of all three variables.

Although there is a relatively large number of moderators compared to the number of clusters in columns 4 and 5, the small sample corrected degrees of freedom were not close to the level where robust-variance estimation starts to perform poorly (Tanner-Smith & Tipton, 2014; Tipton, 2015). None of our moderators has below 10 degrees of freedom in any specification.

The associations between moderators and effect sizes were consistent over specifications in Table 4 in the sense that no moderator changed sign when we added more variables. The Q statistic was reduced the more moderators we added and ranged from 300.2 (column 1) to 184.8 (column 5). These numbers can also be compared to the heterogeneity in the overall short-term effects (Q = 302.8). However, there was still significant heterogeneity and adding moderators did not greatly reduce I ² .

There were few statistically significant moderators, despite some coefficients being relatively large in relation to the baseline effect. Mathematics tests were associated with significantly larger effect sizes than reading tests and QES were associated with significantly larger effect sizes than RCTs in the specifications in columns 3–5 where we included other moderators. The differences are around 0.2 for both moderators. The average effect size in RCTs was still reasonably large and significant in column 2 (ES = 0.17, CI = [0.105, 0.238]).

Two instructional components were associated with larger effect sizes than the baseline effect—progress monitoring and small group instruction—but none of them were statistically significantly different. CAI was associated with smaller effect sizes than the mean and the coefficient was relatively large and significant. Other components with relatively large negative but not statistically significant coefficients were incentives, peer-assisted instruction, and other domain. The coefficients on general tests, grade, duration, and implementation problems were all small and not significant.

The lack of statistical significance in Table 4 does not necessarily mean that there are no differences between the components. In Table 5, we used the specification in column 5 of Table 4 and report the results from tests, which examined if the coefficients of the intervention components were significantly different from each other. There were two significant differences out of 21 tests: small group instruction was associated with significantly larger effect sizes than CAI and incentives.

The specifications in Table 4 and the tests reported in Table 5 are an attempt to get closer to isolating the association between intervention components and effect sizes. However, this approach has its own drawbacks. Importantly, as there are many potentially omitted variables, we want to emphasise that the coefficients in Table 4 represent associations and not causal effects. For example, Table 4 does not contain any interactions among instructional methods and many interventions change more than one instructional method. We could not investigate combinations of instructional methods because no combination of methods was included in enough interventions. For all combinations, the RVE procedure ran into degrees of freedom problems when we ran meta-regressions including just a constant with the outcome variable being the effect sizes from interventions including the combination of interest (i.e., without adding additional moderators).

Adding more moderators may help account for confounding, but by adding moderators that do not explain much variation we may lose statistical power. Including many moderators may moreover cause multicollinearity. There were for example two pairs with a higher than 0.5 correlation in Table 3: QES and CAI (0.50), and incentives and CAI (0.51). Furthermore, multicollinearity does not only arise from pairwise correlations and may therefore be a problem for the other estimates as well. That is, we may lack sufficient variation to get precise estimates of the moderators’ associations with effect sizes in the more comprehensive specifications in columns 4 and 5 of Table 4. On the other hand, we are testing multiple hypotheses and should adjust the p-values accordingly. We did not prespecify a multiple hypothesis testing adjustment but none of the significant differences survive an adjustment for the 21 tests in Table 5 using the procedure suggested by Holm (1979). We would have to allow a false discovery rate of 24% for the test with the lowest p value to be significant using the adjustment procedure suggested by Benjamini and Hochberg (1995).

In sum, our analysis of heterogeneity revealed that almost all intervention components are associated with positive and significant effect sizes in at least one type of analysis (the exception is incentives). However, our evidence of differences in effect sizes between intervention components and our evidence of how components should be combined was relatively weak.

Results of sensitivity analyses

The sections below report results from our sensitivity analyses. We tested whether effect sizes were associated with effect size measurement, adjusting for outliers, adjusting for clustered assignment of treatment, multiply imputing moderators with missing observations, risk of bias, and finally, publication bias. For all but the last analysis, we concentrated on differences between effect sizes based on mathematics and reading, and RCTs and QES. That is, we focused on the two moderators associated with the largest and the most consistently significant differences in effect sizes. The reason was that the main analysis either indicated small(er) differences or did not have enough power to detect differences between other moderators. We used the RVE procedure and estimated, when possible, separate specifications for QES and RCTs, and for effect sizes based on reading and mathematics tests.

Figures 8–11 show the results for effect size measurement, outliers, clustered assignment of treatment, moderators with missing observations, and risk of bias. The baseline average effect size and confidence interval for mathematics tests (Figure 8), reading tests (Figure 9), RCTs (Figure 10) and QES (Figure 11) is shown at the top of each graph, and the changes to the effect size and confidence interval is displayed by the type of sensitivity analysis.

OPEN IN VIEWER

Figure 8

Sensitivity analyses: Effect sizes based on mathematics tests

OPEN IN VIEWER

Figure 9

Sensitivity analyses: Effect sizes based on reading tests

OPEN IN VIEWER

Figure 10

Sensitivity analyses: Effect sizes in randomised controlled trials

OPEN IN VIEWER

Figure 11

Sensitivity analyses: Effect sizes in quasiexperimental study

Publication bias

Figure 12 displays a funnel plot of all short-run effect sizes on the x axis and standard errors on the y axis. The effect sizes were mean-centred, so that the zero-line in the plot represents the mean effect size. There were, as mentioned, a few outliers with very large, positive effect sizes, but we found no outliers with large, negative effect sizes. Apart from these outliers, there were no clear signs of asymmetry. We got similar results when we used study level averages (see Figure 13).

OPEN IN VIEWER

Figure 12

Funnel plot: Mean-centred effect sizes on the x axis and standard errors on the y axis

OPEN IN VIEWER

Figure 13

Funnel plot: Mean-centred study level effect sizes on the x axis and study level standard errors on the y axis

We also performed Egger's test (Egger et al. 1997) and rejected the null hypothesis of no bias (p < .001). However, when we excluded four outliers (study level average g > 1), we could no longer reject the null hypothesis (p = .099). Lastly, we included an indicator for studies published in scientific journals in the baseline meta-regression. There was little evidence that published studies contained larger effect sizes than studies that were not published in journals, the coefficient on the journal indicator was small and not significant (β = .035, CI = [−0.123, 0.192]).

For Egger's test, we used the Stata command metabias and study level effect sizes and standard errors (the command does not perform cluster-robust variance estimation). The number of studies was 69 and the number of student observations included in these studies was 33,772 (i.e., we included all studies except one that only provided follow-up effect sizes and one with an overlapping sample). When we adjusted for outliers, the same numbers were 65 and 33,547, respectively. The number of clusters, effect sizes, and student observations in the estimation testing whether journal publications have larger effect sizes was estimations are 66, 194 and 79,191, respectively.