Protocol for a Systematic Review: Targeted School-Based Interventions for Improving Reading and Mathematics for Students With or At-Risk of Academic Difficulties in Grade 7 to 12: A Systematic Review

Reasons for academic difficulties

Students may be struggling for a number of reasons. However, the latest PISA-tests suggest that students from families with low SES are overrepresented among low performing pupils (OECD 2010, 2013). The reasons for low achievement in general are thus likely connected to the challenges faced by low SES students, a group for which there is a relatively large research literature from different academic fields examining the reasons for why their educational achievement is lower. We discuss this literature below. ¹

Lower innate ability does not seem to be a major explanation of achievement differences. Recent evidence from the US indicates that measures of mental ability do not differ significantly between high and low SES children in the early ages. Tucker-Drob et al. (2011) found no significant differences on tests of infant mental ability at the age of 10 months between children in families with high and low SES. At age two however, children in high SES families scored about one third of a standard deviation higher. Genes accounted for nearly 50 percent of the variation in mental ability of children raised in high SES homes, but only a negligible share of the variation in mental ability of children raised in low SES homes. Similar results were obtained in a follow-up measurement using tests of school readiness (Tucker-Drob et al., 2013). Similarly, the differences in test scores between black and white American children have been found to be about one standard deviation already at age 3.

However, examining infants 8 to 12 months old, Fryer and Levitt (2013) found no significant differences between Hispanics, Asians, Blacks, and Whites. Furthermore, early childhood poverty has been shown to be a better predictor of later cognitive achievement than poverty in middle or late childhood, which is hard to explain by differences in innate abilities (Hackman & Farah, 2009). While hereditary factors cannot be completely ruled out as a determinant of differing educational achievement with current knowledge, these results suggest that the environment is the constraining factor for the achievement of low SES children (Burchinal et al., 2011; Nisbett et al., 2012).

Consistent with the differences between high and low SES children being present early on, the early childhood environment seems to be an important explanation. Currie (2009) surveys a large literature documenting that low SES children have worse health on a very broad range of measures, including fetal conditions, health at birth, incidence of chronic conditions, and mental health problems. Child health problems in turn influence both educational and labour market outcomes, but seem to be smaller for educational outcomes than for earnings (Currie, 2009).

Family resources and the home environment of low SES students also seem less conducive to high educational achievement (Jacob & Ludwig 2008). High SES families on average provide a richer language and literacy environment (Hart & Risley, 2003), use different parenting practices, and spend more money on early childhood education (Esping-Andersson et al., 2012). Low SES parents also seem to have lower academic expectations for their children (Bradley & Corwyn, 2002; Slates et al, 2012), and teachers have lower expectations for low SES students (e.g. Good et al., 2003; Timperley & Phillips, 2003).

The neighbourhoods students grow up in are another potential determinant of achievement. Regarding the relative importance of families and neighbourhoods, the review in Björklund & Salvanes (2011) indicate that family resources are the more important explanatory factor. Results from experiments where families are randomly given the opportunity to change neighbourhoods show mixed results (e.g. Chetty et al., 2015; Kling et al., 2007). But it seems likely that low SES students live in neighbourhoods that are less supportive of high educational achievement in terms of, for example, peer support and role models. To get by in a disadvantaged neighbourhood may also require a very different set of skills compared to what is needed to thrive in school, something which may increase the risk that pupils have trouble decoding the “correct” behaviour in educational environments (Heller, Shah, Guryan, Ludwig, Mullainathan & Pollack, 2015).

In sum, the evidence indicates that large and significant differences are present already well before children start school. Heckman (2006) furthermore argues that schools are not the major source of inequality in student performance, as gaps in test scores across socioeconomic groups are stable from third grade and onwards. School interventions do however have the potential to significantly reduce the gap between high and low SES students (Björklund & Salvanes, 2011).

Theoretical perspectives

The reasons why students may be struggling laid out in the previous section are multifaceted, and the theoretical perspectives underlying interventions are therefore likely to be broad. Nevertheless, we anticipate that three superordinate components will be characteristic for the majority of the included interventions. These components can be abridged to: •

Adaptation of behaviour (social learning theory).

We emphasise that the following presentation of theoretical perspectives is not all-covering, and, though components are presented as demarcated, they contain some conceptual overlap.

Social learning theory has its origins in social and personality psychology, and was initially developed by psychologist Julian Rotter and further developed especially by Albert Bandura (1977; 1986). From the perspective of social learning theory, behaviour and skills are primarily learned by observing and imitating the actions of others, and behaviour is in turn regulated by the recognition of those actions by others (reinforcement), or discouraged by lack of recognition or sanctions (punishment). According to social learning theory, creating the right social context for the student can therefore stimulate more productive behaviour through social modelling and reinforcement of certain behaviours that can lead to higher educational achievement.

Cognitive developmental theory is not one particular theory, but rather a myriad of theories about human development that focus on how cognitive functions such as language skills, comprehension, memory and problem-solving skills enable students to think, act and learn in their social environment. Some theories emphasize a concept of intelligence where children gradually come to acquire, construct, and use cognitive functions as the child naturally matures with age (e.g. Piaget, 2001; Perry, 1999). Other theories hold a more socio-cultural view of cognitive development and use a more culturally distinct and individualized concept of intelligence that to a greater extent includes social interaction and individual experience as the basis for cognitive development. Examples include the theories of Robert Sternberg (2009) and Howard Gardner (1999).

Pedagogical theory draws on the different disciplines in psychology and social theory such as cognitivism, social-interactional theory and socio-cultural theory of learning and development. There is not one uniform pedagogical model, but examples of contemporary models in mainstream pedagogy are concepts such as Scaffolding (Bruner, 2006) and the Zone of Proximal Development (Vygotsky, 1978), which origins in developmental and educational psychology. These notions hold that learning and development emerge through practical activity and interaction. Acquisition of new knowledge is therefore considered to be dependent on social experience and previous learning, as well as the availability and type of instruction. Accordingly, school interventions require educators to interact and organise the learning environment for the student in certain ways to fit the individual student's needs and potentials for development.

Interventions in practice

In general, school interventions affect academic achievement by changing the methods by which instruction is given (instructional methods), or by changing the content of what is taught (the content domain), and many combine several intervention components as well as theoretical perspectives. Previous reviews (e.g. Dietrichson et al. 2015a) indicate that we will for example find interventions using the following categories of instructional methods: tutoring, coaching/mentoring, cooperative learning/peer-assisted learning, computer-assisted instruction, feedback and progress monitoring, behavioural/psychological interventions, and incentive programs. Reading interventions directed to older students often target content domains such comprehension, fluency, word study, and vocabulary (e.g. Scammaca et al. 2015). Slavin et al. (2009) compared curricula for middle and high school mathematics that differed over for example how much they emphasised domains such as problem solving and conceptual understanding. Gersten et al. (2009) used the following domains to divide mathematics interventions into categories: (a) operations (e.g. addition, subtraction, and multiplication), (b) word problems, (c) fractions, (d) algebra, and (e) general math proficiency (or multiple components).

Earlier research has shown that very different types of academic interventions can improve academic performance, both across methods, delivery mode, age group and duration (e.g. Cheung & Slavin, 2012; Dietrichson et al. 2015a). Both reading strategy instruction and peer-mediated learning programmes such as paired reading have been shown to be effective in improving literacy skills of struggling secondary school readers. These are two types of programmes that clearly have different components and delivery modes (Edmonds, Vaughn, Wexler, Reutebuch, Cable, Klingler Tackett & Wick Schnakenberg, 2009). In another example, Good, Aronson, and Inzlicht (2003) show that changing expectations of seventh grade students at risk for stereotype-based underperformance (minority and low-income students in general, and girls regarding mathematics) can improve standardised test scores.

On the other hand, while some interventions which rely on a specific approach may prove effective, other interventions relying on a similar approach may not. As an example, computer-assisted instruction programmes range from strong effects to no effects at all on mathematical achievement (Kulik, 2003; Chambers, 2003; Cheung & Slavin, 2013), and while computer-based instruction programmes overall show some effect on math skills, they seem to have smaller impact on reading skills (Kulik, 2003; Cheung & Slavin, 2012).

There are indications that one-on-one tutoring and small group tutoring have some of the largest effects on academic outcomes across conditions in both reading and mathematics. However, the evidence base varies across interventions, and in general there have been more studies examining reading interventions than math interventions (e.g. Cohen, Kulik, & Kulik, 1982, Dietrichson et al., 2015a; Flynn, Marquis, Paquet, Peeke, & Aubry, 2012; Forsman & Vinnerljung, 2012; Reisner, Petry, & Armitage, 1989, 1990; Robinson et al., 2005). Furthermore, recent research also demonstrates that peer-mediated interventions such as collaborative learning interventions and peer-tutoring in general have promising effects for disadvantaged and low performing secondary school students (McMaster & Fuchs, 2002; Bowman-Perrott et al., 2013).

This outlined research points to direct and individual instruction, small or one-on-one defined settings, and mediation from adults or more competent peers as being especially important for struggling learners. Furthermore, interventions such as tutoring and structured peer-mediated interventions often have in common that they comprise an eclectic theoretical model that combines components from all three perspectives on learning presented in the previous section. They are comprehensive interventions that relies on a complex of mechanisms such as increased feedback and tailor-made instruction (pedagogical theory), regulation of behaviour by for example rewards or interaction with role models (social learning theory), and development of cognitive functions such as learning how to learn (cognitive developmental theory).

Another way of viewing these and other types of interventions is that they address the differential family and neighbourhood resources of high and low SES students described in the previous section. Students from high SES families are likely to have access to “tutors” all year round, as parents, siblings and other family members help out with homework and schoolwork. Interventions to change mindsets, increase expectations, and mitigate stereotype threat also substitute for high SES families and teachers already having such expectations or teaching their children such a mindset. Different types of extrinsic rewards may be a way to bolster motivation, which may be especially important for students whose families place less weight on educational achievement.

Furthermore, if, as indicated in the previous section, the differences between high and low SES students and students with academic difficulties can be understood as a consequence of differential access to a combination of resources, remedial efforts may need to address several problems at once to be effective. Programs that combine certain components may therefore be more effective than others. To exemplify, both programmes deemed to be backed by strong evidence of effectiveness in improving middle and high school mathematics in Slavin et al. (2009) include several components. The first, Student Teams-Achievement Divisions, includes learning in small teams, individual assessments and accountability, as well as rewards based on team performance. The second, IMPROVE, combines cooperative learning, metacognitive instruction, and mastery learning. A further example highlights that it does not have to be just academic problems that affect school achievement. Two recent studies examine the programme Becoming A Man, which includes features from cognitive behavioural therapy and the development of social-cognitive skills such as generating new solutions to problems, learning new ways of behaving, considering another's perspective, thinking ahead, and evaluating consequences ahead of time. The program significantly reduced instances of violent-crime and decreased dropout rates, but did not increase test scores in a randomised field experiment including 2,740 male youth in grades 7-10 from high-crime and high-poverty Chicago neighbourhoods (Heller, Pollack, Ander & Ludwig, 2013). However, combined with a math tutoring intervention, the programme also significantly increased standardized test scores in an experiment with a population of 106 males from similar neighbourhoods (Cook, Dodge, Farkas, Fryer, Guryan, Ludwig, Mayer, Pollack & Steinberg, 2014).

Another reason why it is interesting to examine combinations of components relates to an often suggested explanation for missing impacts: lack of motivation among participants (e.g. Fuchs, Fuchs & Kazdan, 1999; Edmonds et al. 2009). It is therefore possible that interventions will be more effective if they also include some form of rewards for participating students and implementing teachers, along with other components providing for instance specific pedagogical support. At the same time, just providing motivation or incentives may not be enough. For example, in a large scale randomised experiment (in total the experiment involved around 27,000 students) second graders were paid to read books, fourth and seventh grade students were paid for performance on a series of assessments, and ninth graders were paid for grades. None of these treatments yielded significant effects on the aggregate treatment level (Fryer, 2011).

For struggling students in grades 7-12, who are likely to have a history of low achievement, finding the right combination of intervention components may be especially pertinent (e.g. Fuchs et al. 1999; Edmonds et al. 2009). Some researchers have recommended, based on the perceived low relative cost-effectiveness of interventions directed to adolescents, that resources should disproportionally be used for early interventions (e.g. Esping-Andersen, 2004, Heckman, 2006), or that secondary schools should primarily be providing technical and vocational training for disadvantaged teenagers (Cullen, Levitt, Robertson, & Sadoff, 2013). However, Cook et al. (2014) argued that the low relative cost-effectiveness may be a premature conjecture, as previous interventions for youths have often not combined the fostering of academic skills with other important factors for academic success, such as social-cognitive (or non-cognitive) skills. As for example social information processing programmes (Wilson & Lipsey, 2006a; 2006b), and programmes based on cognitive behavioural therapy (e.g. Lipsey, Landenberger & Wilson, 2007) have been found to effectively reduce problematic behaviour and promote social-cognitive skills, combinations with more academically oriented interventions look promising.

Prior reviews

In some regards, this review shares common ground with existing Campbell reviews and reviews in progress such as “Impacts of After-School Programs on Student Outcomes: A Systematic Review” (Zief, Lauver, & Maynard, 2006), “Dropout Prevention and Intervention Programs: Effects on School Completion and Dropout among School-aged Children and Youth” (Wilson, Tanner-Smith, Lipsey, Steinka-Fry, & Morrison, 2011), and “Effects of College Access Programs on College Readiness and Enrollment” (Harvill, Maynard, Nguyen, Robertson-Kraft, Tognatta, & Fester, 2012). ²

Nevertheless, this review differs in substantial ways from these existing Campbell reviews. First, with the exception of Zief et al. (2006), the listed reviews do not explicitly target an educationally disadvantaged or low performing student population. Zief et al. (2006) on the other hand excluded interventions performed outside North America, and three of the five studies included were of programmes primarily designed to reduce negative behaviours such as delinquency and drug use; i.e. the programmes did not target academic achievement as their primary outcome. Wilson et al. (2011) did not explicitly target students with or at-risk of academic difficulties, many of the studies in their review of dropout prevention and interventions programmes of course included at-risk groups. Except their review, existing Campbell reviews all focus on one specific type of intervention or setting. A major difference between their review and the current proposal is that they focused on programmes of school completion and dropout prevention, and outcome measures as dropout and graduation rates. This review will only include studies that report results on standardised tests in reading and mathematics. There is some overlap between the types of interventions included but also clear differences, as many of the interventions we will include do not target dropout and interventions such as for example paid employment for students, community service programs, and vocational training will not feature in our review.

In addition to these Campbell reviews and reviews in progress, there are other related reviews with a similar broad scope and a target group overlapping ours to some degree. ³ Slavin et al. (2009) reviewed programmes in middle and high school mathematics, whereas Slavin, Cheung, Groff, & Lake (2008) reviewed reading programmes for middle and high schools. However, these reviews focused on all kinds of programmes, not programmes for at-risk or low-performing students. Furthermore, Wanzek, Vaughn, Wexler, Swanson, Edmonds & Kim (2006) reviewed reading interventions directed to students in grades K-12 with learning disabilities, and Edmonds et al. (2009), Flynn, Zheng & Swanson (2012), and Scammaca et al. (2015) reviewed interventions for struggling readers in grades 6-12, 5-9, and 4-12, respectively. ⁴ These reviews thus covered low achieving students, but neither at-risk students nor areas other than reading. Gersten et al. (2009) examined four types of components of mathematics instruction for students with learning disabilities, but did not include studies for students at-risk (or more general reasons for low performance than learning disabilities). Dietrichson et al. (2015a) on the other hand included studies in both reading and mathematics and based inclusion on the share of students with low SES, but did not consider whether students had academic difficulties or not.

In terms of findings related to this review's primary outcome measures, the reviews that have focused on the effects of academic interventions on reading test scores all showed positive overall effect sizes, although there was a rather large variation between interventions in all reviews (Edmonds et al., 2009; Flynn et al., 2012; Scammaca et al., 2015; Slavin et al., 2008; Slavin et al., 2009; Wanzek et al., 2006). The four reviews of reading interventions directed to struggling readers reported positive effects in general but few reliable differences over types of interventions (Edmonds et al., 2009; Flynn et al., 2012; Scammaca et al., 2015; Wanzek et al., 2006). An exception is that reading comprehension interventions were associated with significantly higher effect sizes than fluency interventions in Scammaca et al. (2015), but this difference disappears when only standardised measures were considered.

Gersten et al. (2009) examined four components of mathematics instruction for students with learning disabilities, and found most support for approaches to instruction (e.g. explicit instruction, use of heuristics) and/or curriculum design, and providing formative assessment data and feedback to teachers. Dietrichson et al. (2015a) examined interventions that have used standardised tests in reading and mathematics and categorise 14 intervention components mainly delimited by the instructional methods used. Tutoring, feedback and progress monitoring, and cooperative learning have the largest and most robust average effect sizes.

The best evidence syntheses by Slavin et al. (2008) and Slavin et al. (2009) both point to instructional-process programmes, especially programmes that incorporate cooperative learning, as having larger effects than curricula based interventions, and computer assisted instruction programmes. Slavin et al. (2009) found no indications that effect sizes differ between socioeconomically disadvantaged students and non-disadvantaged students. However, only a relatively small subset of studies reported results differentiated by SES, and the review does not contain information about whether the programmes that in general show the largest effect sizes also has the largest effect sizes for disadvantaged students.

Slavin et al. (2009) and Edmonds et al. (2009) reported that some programmes, which have been shown to be effective for younger students, may have smaller or no effects for older students. Effect sizes were smaller for older students also in Scammaca et al. (2015), although not significantly different. As discussed in the previous section, there are also other indications that earlier interventions are more cost-effective, but, as argued in Cook et al. (2014), this may be because programmes directed to older target groups often have lacked components that are especially important for older students. Neither the question of whether interventions are less effective for older students, nor whether combinations of components are important is settled in the reviews covered in this section.

The contribution of this review

Academic difficulties and lack of educational attainment are significant societal problems, and special education is challenging and costly, not least because research on ability grouping indicates that grouping students based on prior displayed abilities or subjective expectations about their abilities might have the unintended consequence of reproducing social inequalities in educational attainment (Condron, 2008; Gamoran 2004; Hattie 2002; Justice, Petscher, Schatschneider, & Mashburn, 2011; Kerckhoff 1993; Lubbers, Snijders, & Van Der Werf, 2011; Schofield 2010; Van de Werfhorst & Mijs 2010). Moreover, as shown by the Salamanca declaration from 1994 (UNESCO, 1994), there has for decades been a great interest among policy makers to improve the inclusion of students with academic difficulties in mainstream schooling, and a desire to increase the number of empirically supported interventions for these student groups.

The main objective of this review is to provide policy makers and educational decision-makers at all levels – from governments to teachers – with evidence of the effectiveness of interventions aimed to improve the results of students with or at-risk of academic difficulties. To achieve this objective we will compare the effects of interventions that differ in terms of their components regarding both instructional methods and the content taught. To be specific, we are interested in providing evidence on whether for example tutoring improves educational achievement. However, we would also like to examine whether tutoring interventions improve educational achievement more than, say, cooperative learning interventions, and if interventions work better in mathematics than in reading, or when they emphasize vocabulary rather than fluency. Furthermore, it is presently not known whether interventions that combine components, for example cooperative learning combined with a component that gives teachers and students frequent feedback on student progress, or tutoring combined with socio-emotional training, are more effective than interventions that use only a single component.

To this end, we have chosen a broad scope in terms of the target group and the types of interventions we include. We will also include interventions where the effects are measured by standardised tests in reading and mathematics. The reason is that many interventions are not directed specifically to either subject and outcomes are therefore often measured in both (Dietrichson et al. 2015a). Earlier reviews of interventions to reasonably similar target groups (e.g. Gersten et al. 2009, Slavin et al. 2011, Dietrichson et al. 2015a) provide tentative evidence that similar types of interventions are effective for both struggling and low SES students, but more knowledge about whether this is so would be welcome. That this knowledge is not complete is a reason to keep both the types of interventions we include and the target group relatively broad. Including both students with and at-risk of academic difficulties in the target group should also decrease the risk of biasing the results due to omission of studies where information about either academic difficulties or at-risk status is available, but not both. Furthermore, making comparisons over intervention components such as instructional methods and content domains within one review, rather than across reviews, should increase the possibilities of a fair comparison. For instance, controlling that effect sizes are calculated in the same way, that the definitions of intervention components are consistent, and that moderators are coded in the same way, is easier within the scope of one review.

In isolation, this last argument suggests that all interventions aiming to improve educational achievement for our target population should be included. However, we also want to explore why certain interventions work better than others. The results in the reviews of for example Slavin et al. (2008, 2009) and Dietrichson et al. (2015a) point to substantial variation in effect sizes aimed to improve test scores in reading and mathematics. Importantly, this variation is also found within types of interventions. For the exploration of variation in effect sizes, a broad scope may turn into a disadvantage, as information about moderators that are important in order to explain variation for some types of interventions are not relevant for others. We have therefore delimited the included interventions to those that are targeted, rather than universal, and performed in a regular school situation during the regular school year. This delimitation increases the probability that potentially important moderators, such as dosage are reported in a comparable way.

Hopefully, the review should therefore be able to provide guidance about what components of interventions, and combinations of components, that are effective. Earlier reviews with a comparable focus have either not included intervention components together with other moderators in a meta-regression, or only included broad categories of interventions. For example, reviews have coded interventions over contrasts between treatment and control groups regarding the instructional methods used, or regarding the type of content taught, but not both (e.g. Dietrichson et al., 2015a; Gersten et al., 2009; Scammaca et al., 2015). Thus, the first risks confounding the effects of intervention components with for example participant characteristics, and the second risks confounding methods with content.

Types of interventions

For intervention studies to be included in the review it must be clear that the intervention is structured so that it works to improve academic achievement or specific academic skills. This does not mean that the intervention must consist of academic activities, but rather that the explicit expectation must be that the intervention, regardless of the nature of the intervention content, will result in improved academic performance or a higher skill level in a specific academic task. Furthermore, an explicit academic aim of the intervention does not per se exclude interventions that also include non-academic objectives and outcomes.

Interventions without academic outcomes or interventions having academic learning as a possible secondary goal (such as interventions where behavioural or socioemotional problems is the primary intervention aim, like Classroom Management or Families and Schools Together) will be excluded. However, interventions with behavioural and socioemotional components may very well have academic achievement as one of their primary aims (e.g. some mindset and stereotype threat interventions). Such interventions will be included if this aim is made explicit in the study (and the outcomes are measured by standardised tests in reading or mathematics, see below section Types of outcome measures for more details).

Furthermore, we will only include school-based interventions; that is, interventions performed in schools during the regular school year, and schools are one of the stakeholders. Judging by the results in the related review of Dietrichson et al. (2015a), this restriction excludes summer reading programs and some after-school programs (which may, but need not, be performed outside of school by other actors). Both of these types of interventions appear to be using qualitatively different components compared to interventions performed in school. They are often also different in terms of for example who deliver them, how different aspects of intervention dosage are measured, and whether and how implementation is assessed. In addition, there is a very recent review of summer reading programs (Kim & Quinn, 2013), and one earlier Campbell review of after-school programs (Zief et al., 2006). Our criteria would also exclude for example parent tutoring programmes and other programmes delivered in the home of students. If interventions are mainly delivered in school during the school year, but also include a component delivered outside of school, they will be included.

Besides having as their explicit primary expectation that the intervention will improve the academic performance of the student, eligible interventions for review must also be targeted (or selected/indicated). That is, interventions which, in contrast to universal interventions, are aimed at certain students and/or student groups identified as having academic difficulties, or being at-risk of such difficulties (see below for a detailed description of the types of participants we will include).

Universal interventions, applied to improve the quality of the common learning environment at the school level in order to raise academic achievement of all students (including average and above average students), will be excluded. Interventions such as the one described in Fryer (2014) where a bundle of best practices are implemented at the school level in low achieving schools, where most or possibly all students are struggling or at risk, will therefore be excluded. This criteria also excludes whole-school reform strategy concepts such as Success for All, curriculum-based programmes like Elements of Mathematics (EMP), as well as reduced class size interventions. It also excludes interventions where teachers or principals receive professional development training in order to improve general teaching or management skills. Interventions targeting students with or at-risk of academic difficulties may on the other hand include a professional development component, for example when a reading programme includes providing teachers with reading coaches. Such interventions will be included.

Types of participants

The population samples eligible for the review include students attending regular schools in grades 7-12, who are having academic difficulties, or are at-risk of such difficulties. Students attending regular private, public, and boarding schools are included, and students receiving special education services within these school settings are also included. Grades 7-12 corresponds roughly to secondary school, defined as the second step in a three-tier educational system consisting of primary education, secondary education and tertiary or higher education. The number of years a child attend secondary schooling varies across the OECD countries, though most often secondary schooling is grades 7-12 or 10-12. The former is the case for instance in France, Spain, Japan, UK, and most parts of Australia, and the second is the case for school systems in countries such as Italy, Turkey, Sweden and Denmark. We will include studies with a student population younger than 7-12 as long as the majority of the students are in grades 7-12. The age range included will also differ between countries, and sometimes between states within countries. Typically, ages will range from 12-14 to 17-19.

The eligible student population includes both students identified in the studies by their observed academic achievement (e.g., low academic test results, low grade point average or students with specific academic difficulties such as learning disabilities), and students that have been identified primarily on the basis of their educational, psychological, or social background (e.g., students from families with low socioeconomic status, students placed in care, students from diverse ethnic/cultural backgrounds, and second language learners). We will however exclude interventions targeting students with physical learning disabilities (e.g. blind students), students with dyslexia/ dyscalculia, and interventions that are specifically directed towards students with a certain neuropsychiatric disorder (e.g. autism, ADHD), as these interventions are probably very different from interventions targeting the general struggling or at-risk student population.

We believe it is important to include students that for other reasons are struggling together with groups that are deemed at-risk, or are considered educationally disadvantaged. There is substantial overlap between these groups in the studies we have found in a previous review (Dietrichson et al. 2015a). A motivating example comes from studies that target a high poverty area, and then randomly select a number of students with test scores below a certain level in each school that receive the intervention. These students are thus likely to be low SES, but information about SES is not always included. That is, shares of low SES students are only reported on the school or district level, and sometimes not at all. A second example would be studies that target low performing schools, and then perform an intervention for the sub-group of low SES students. In this case, low SES students are likely to be struggling, although this information is not always included.

Thus, choosing to include only studies that examine either students with academic difficulties or low SES students may exclude studies that in all likelihood target the same student population. We think that the risk of biasing our results by such a choice is larger than the possible comparison problems arising from including both students with academic difficulties and low SES students. A similar case can be made for other at-risk groups, for example students from diverse ethnic/cultural backgrounds, which in many cases overlap with low SES students.

Finally, there are also good reasons to suspect a substantial overlap of the reasons for why these groups need interventions. While the earlier literature has not fully converged on a ranking of these reasons, the differential access to family resources is a major contributor to these groups' educational disadvantage; something which school-based interventions may compensate for. The reasons for low performance are thus likely connected to the challenges faced by at-risk students.

Some interventions may include other students, who are neither with nor at-risk of academic difficulties. An example may be a cooperative learning intervention where high performing students are paired with struggling students. Studies of such interventions will be included if the total sample (treatment and control group) include at least 50% students that are either having academic difficulties or are at-risk of developing such difficulties.

Types of outcome measures

As the overall purpose of the review is to evaluate evidence on effects of educational interventions on academic achievement, we will include outcomes that cover two main areas of fundamental academic skills: •

Standardised tests in reading

Studies will only be included if they consider one or more of the primary outcomes. As standardised tests, we will consider norm-referenced tests (e.g. Gates-MacGinitie Reading Tests and Star Math), state-wide tests (e.g. Iowa Test of Basic Skills), and national tests (e.g. National Assessment of Educational Progress). If it is not clear from the description of outcome measures in the studies, we will use electronic sources to determine whether a test is standardised or not. For example, if a commercial test has been normed, this is typically mentioned on the publisher's homepage. If there is no such mention, we will consider the test as being not standardised.

We restrict our attention to standardised tests in part to increase the comparability between effect sizes. Earlier related reviews of academic interventions have pointed out that effect sizes tend to be significantly lower for standardised tests compared to researcher-developed tests (e.g. Flynn et al., 2012; Gersten et al., 2009; Scammaca et al., 2015). Scammaca et al. (2015) furthermore reported that whereas mean effect sizes differed significantly between the periods 1980-2004 and 2005-2011 for other types of tests, mean effect sizes were not significantly different for standardised tests. As researcher developed tests are usually less comprehensive and more likely to measure aspects of content inherent to treatment but not control group instruction (Slavin & Madden, 2011), standardised tests should provide a more reliable measure of lasting differences between treatment and control groups. For this reason, we will not consider tests where researchers have picked a subset of questions from a norm-referenced test as being standardised. In sum, while researcher developed tests may be highly useful for certain purposes (e.g. testing specific intervention mechanisms), we believe they would be less useful for the purposes of this review.

We will include tests of specific domains (e.g. vocabulary, fractions) as well as more general tests, which test several domains of a subject. Tests of subdomains have significantly larger effect sizes compared to more general tests in Dietrichson et al. (2015a). This result may indicate that interventions often target certain domains and not general performance skills, or that it may be easier to improve scores on tests of subdomains than on tests of more general skills, or that tests of subdomains may be more likely to be inherent to treatment (see Slavin & Madden, 2011 for a discussion of the latter). At the same time it seems reasonable that interventions that target subdomains of reading and mathematics be tested on whether they affect these subdomains. Therefore, we do not want to exclude either type of test, but will code the type of test, as well as the content domain of the intervention and use the type of test as a variable in the moderator analyses.

Based on findings in Dietrichson et al. (2015), we expect that a large majority of studies only have reported outcomes of tests performed within 3 months after the end of intervention. We will consider longer run outcomes as well, if they are available (see section Multiple time points below).

There are many other important outcome measures that we do not include (e.g. grades, dropout, and uptake of secondary/tertiary education). We make this choice to streamline the review, and to increase comparability across contexts. Grade setting and the presence of certain education options (e.g. vocational training tracks) are likely to differ more across school systems and countries than standardised tests.

Types of study designs

Types of studies included are studies that use a treatment-control group design or a comparison group design, and adequately address the subject of effectiveness of interventions to improve the students' academic achievement: RCTs, including cluster-RCTs; QRCTs, i.e., where participants are allocated by means such as alternate allocation, person's birth date, the date of the week or month, case number, or alphabetical order; and QES. To be included, QES must credibly demonstrate that outcome differences between treatment and control groups is the effect of the intervention and not the result of systematic baseline differences between groups. That is, selection bias should not be driving the results. This assessment is included as a part of the risk of bias tool, which we elaborate on in section Risk of bias, and in Appendix C.

A control group is defined as a non-treatment condition; a comparison group is defined as an alternative treatment condition. Eligible types of control groups include waitlist controls and no-treatment controls. However, in this review the waitlist controls and no-treatment controls only differ in the time frame in which researchers can follow the differences between groups because students in both waitlist and no-treatment controls are offered regular schooling by default.

Comparison designs compare alternative treatments against each other. Comparison designs will be analysed separately from treatment-control designs. We elaborate in section Synthesis procedures and statistical analysis on how we will use comparison designs. Studies using single group pre-post comparison will not be included. Effect sizes from such studies are not comparable to effect sizes from treatment-control designs if, for example, there is progression in students' knowledge over time, which is typically the case.

Duration of interventions

There will be no initial criteria for duration of interventions, but the duration of included interventions will be coded for the review.

Types of settings

Only studies carried out in OECD countries will be included. This selection is conducted to ensure a certain degree of comparability between school settings to align treatment as usual conditions in included studies. For similar reasons we will only include studies published in or after 1980. Due to language restrictions, we will only include studies written in English, German, Danish, Norwegian, and Swedish.

Electronic databases

Relevant studies will be identified through electronic searches of bibliographic databases, government and policy databanks. The following bibliographic databases will be searched: •

Academic Search Premier

Search terms

An example of the search strategy for ERIC searched through the Ovid platform is listed below. This strategy will be modified for the different databases. We will report details of the modifications used for other databases in the completed review. The strategy contains also terms on primary school, since the search also will contribute to a review about this younger age group. There may be overlap in the literature among the age groups, and in order to rationalize and accelerate the screening process, we have decided upon performing one extensive search. 1.

(Underachiev* or Under n1 achiev* or lowachiev* or low n1 achiev* or Low N1 perform* or lowperform* or (at-risk or at N1 risk)) N1 (student* or pupil*) or ((high-risk or high N1 risk) N1 (student* or pupil*)) or ((Special N1 Need*) N1 (Student* or pupil*)) or ((Low N1 income) N1 (student* or pupil*))

Searching other resources

The review authors will check reference lists of other relevant reviews and included primary studies for new leads. Citation searching in the Web of Science will also be considered. We will contact international experts to identify unpublished and ongoing studies, and provide them with the inclusion criteria for the review along with the list of included studies, asking for any other published, unpublished or ongoing studies relevant for the review. We will primarily contact corresponding authors of the related reviews mentioned in the section Prior reviews, but extend the contacts to others if we find references to or mentions of ongoing studies in screened publications. We will also search two trial registries: The Institute for Education Sciences' Registry of Randomized Controlled Trials (http://ies.ed.gov/ncee/wwc/references/registries/index.aspx), and American Economic Association's RCT Registry (https://www.socialscienceregistry.org).

Handsearch

The following international journals will be hand searched for relevant studies: •

American Educational Research Journal

The search will be performed on editions from 2015 to review submission of the journals mentioned, in order to capture any relevant studies recently published and therefore not captured in the systematic search.

Grey literature

Additional searches will be made by means of Google and Google Scholar and we will check the first 150 hits. OpenGrey (http://www.opengrey.eu/) will also be used to search for European grey literature. Copies of relevant documents will be made and we will record the exact URL and date of access for each relevant document. In addition we will search the following sites: •

What Works Clearinghouse - U.S. Department of Education, http://www.whatworks.ed.gov .

Risk of bias judgement items

The risk of bias model used in this review is based on nine items (see Appendix C for a fuller description). The nine items refer to: Sequence generation, allocation concealment, blinding, incomplete outcome data, selective outcome reporting, other potential threats to validity, a priori protocol, a priori analysis plan, and confounders (for non-randomised studies).

Confounding

An important part of the risk of bias assessment of effect estimates in non-randomised studies is how studies deal with confounding factors. Selection bias is understood as systematic baseline differences between groups and can therefore compromise comparability between groups. Baseline differences can be observable (e.g. age and gender) and unobservable to the researcher (e.g. motivation). Included studies use for example matching, difference-in-differences, and statistical controls to mitigate selection bias, or demonstrate evidence of pre-treatment equivalence on key risk variables and participant characteristics. In each study, we will assess whether confounding factors have been considered. Furthermore, we will assess how each study deals with unobservables.

There is no single non-randomised study design that always deals adequately with the selection problem. Different designs represent different approaches to dealing with selection problems under different assumptions and require different types of data. There can be particularly great variations in how different designs deal with selection on unobservables. For example, differences in pre-treatment test score levels do not have to be a problem in a difference-in-differences design, where the main identifying assumption is that the trends of the outcome variable in the treatment and control group would not have differed, had the intervention not occurred. Similar differences in levels would, in general, be more problematic in a matching design as they indicate that the matching technique has not been able to balance the sample even on observable variables. For this reason, we will not specify thresholds in terms of pre-treatment differences (in say, effect sizes) for when a study has too high risk of bias on confounding. Each QES will be assessed in terms of the risk that the effect of the intervention is being confounded with observed and unobserved variables.

Importance of pre-specified confounding factors

The motivation for focusing on age and grade level, performance at baseline, gender, socioeconomic background and local education spending is given below.

Development of cognitive functions relating to school performance and learning are age dependent, and furthermore systematic differences in performance level often refer to systematic differences in preconditions for further development and learning of both cognitive and social character (Piaget, 2001; Vygotsky, 1978). Therefore, to be sure that an effect estimate is a result from a comparison of groups with no systematic baseline differences it is important to control for the students' grade level (or age).

Performance at baseline is generally a very strong predictor of post-test scores (Hedges & Hedberg, 2007), and controlling for this confounder is therefore highly important.

With respect to gender it is well-known that there exist gender differences in school performance (Holmlund & Sund, 2005). Girls outperform boys with respect to reading and boys outperform boys with respect to mathematics (Stoet & Geary, 2013), although part of the literature finds that these gender differences vanished over time (Hyde, Fennema, & Lamon, 1990; Hyde & Linn, 1988). As there is no consensus around the disappearance of gender differences, we find it important to include this potential confounder.

Students from more advantaged socioeconomic backgrounds on average begin school better prepared to learn and receive greater support from their parents during their schooling years (Ehrenberg et al., 2001, Fryer & Levitt, 2013). As outlined in the background section, students with socio-economically disadvantaged backgrounds have lower test scores on international tests (OECD, 2010, 2013). Therefore, the accuracy of the estimated effects of an intervention may depend on how well socioeconomic background is controlled for. Socioeconomic background factors are, e.g. parents' educational level, family income, ethnic/cultural background, etc.

Bias assessment in practice

At least two, review authors will independently assess the risk of bias for each included study. Disagreements will be resolved by discussion. We will report the risk of bias assessment in risk of bias tables for each included study.

In accordance with Cochrane and Campbell methods we will not aggregate the 5-point scale across items. Effect sizes given a rating of 5 on any item will not be included in the meta-analysis (the items with a three-point scale do not warrant exclusion). We will only give 5 points for an item to denote a very high risk of bias. A stark example would be a study with 100% attrition in the comparison group and no follow up data. Effect sizes from this study would receive 5 points on incomplete outcome data. This study would not be included in the meta-analysis as it in effect has become a single group study with pre- and post- measures for the experimental group only. Further examples may be QES, which have not controlled for any confounders or not reported the balance on any pre-treatment tests, or studies that completely confound treatment with other effects. An example of the latter is when treatment is assigned on school level and there is one treated school and one control school. Treatment is then completely confounded with school effects. For studies with a lower than 5-point rating, we will use the ratings of the major items in sensitivity analyses.

Effect sizes using continuous data

For continuous data, standardized mean differences (SMDs) will be calculated when means and standard deviations are available. We will use Hedges' g to estimate SMDs where scales have been used to measure the same outcomes in different ways. Hedges' g and its standard error are calculated as (Lipsey & Wilson, 2001:47-49):

g = ( 1 − 1 4 N − 9 ) × ( X 1 ¯ − X 2 ¯ S p )

1

S E g = N n 1 n 2 + g 2 2 N

2

X ¯

s p = ( n 1 − 1 ) s 1 2 + ( n 2 − 1 ) s 2 2 ( n 1 − 1 ) + ( n 1 − 1 ) .

3

Here, s ₁ and s ₂ denotes the standard deviation of the treatment and control group. We will use covariate adjusted means whenever available, and the unadjusted standard deviation. We will use intention-to-treat (ITT) estimates of the mean difference whenever possible, and test whether results are sensitive to the inclusion of treatment-on-the-treated (TOT) or local average treatment effects (LATE). If there is a mix of studies with some reporting change scores and others reporting final values, we will contact the trial investigators and request the final values. If these are unobtainable, we will analyse change scores and final values separately.

Effect sizes using discrete data

Based on findings in Dietrichson et al. (2015a), where only two out of 101 included studies exclusively reported discrete outcome measures, we expect that almost all studies in this literature use continuous outcome measures. We therefore expect to use the methods described in Sanchez-Meca, Marin-Martines, & Chacon-Moscoso (2003) to transform any dichotomous outcomes into SMDs.

Should we find a large enough number of studies using dichotomous outcomes, we will test whether our results are sensitive to combining dichotomous and continuous outcome measures. If this is the case, we will also perform a sensitivity analysis using only dichotomous measures, and the following procedure to calculate effect sizes: We will use the natural logarithm of odds ratios (LOR) in the calculations, together with 95% confidence intervals and p-values, and then convert the results back to the original odds ratios once the meta-analysis is performed. The LOR and its approximate standard deviation are calculated as (Lipsey & Wilson, 2001:53-54):

L O R = log ( a d b c )

4

S E L O R = 1 a + 1 b + 1 c + 1 d

5

Outliers

We will examine the distributions of effect sizes for each outcome category for the presence of outliers. If outliers are found, we will examine the sensitivity of the results by methods suggested by Lipsey & Wilson (2001): trimming the distribution by dropping the outliers and by Windsorizing the outliers to the nearest non-outlier value.

Dealing with missing data

Missing data and attrition rates in the individual studies will be assessed using the risk of bias tool. Studies must permit calculation of a numeric effect size for the outcomes to be eligible for inclusion in the meta-analysis. Where studies have missing summary data, such as missing standard deviations, we will derive these where possible from e.g., F-ratios, t-values, chi-squared values and correlation coefficients using the methods suggested by Lipsey & Wilson (2001). If these statistics are also missing, the review authors will request information from the study investigators. ⁵

If missing summary data necessary for the calculation of effect sizes cannot be derived or retrieved, the study results will be reported in as much detail as possible, i.e. the study will be included in the review but excluded from the meta-analysis. If data is missing regarding moderators, we will use methods for multiple imputation in order to not bias our results by excluding these studies (see e.g. Rubin (1996) and Pigott (2009) for why leaving out studies/effect sizes with missing values normally yields biased estimates). We will use the Stata command mi impute with sequential imputation using chained equations to generate values for missing observations. All variables without missing observations will be used in the estimation to impute values for variables with missing observations.

Clustered assignment of treatment

Errors in statistical analysis can occur when the unit of allocation differs from the unit of analysis. In cluster randomized trials, participants are randomized to treatment and control groups in clusters, either when data from multiple participants in a setting are included (creating a cluster within the school or community setting), or when participants are randomized by treatment locality or school. QES may also include clustered assignment of treatment. Effect sizes and standard errors from such studies may be biased if the unit-of-analysis is the individual and an appropriate cluster adjustment is not used (Higgins & Green, 2011).

If possible, we will adjust effect sizes individually using the methods suggested by Hedges (2007) and information about the intra-cluster correlation coefficient (ICC), realized cluster sizes, and/or estimates of the within and between variances of clusters. If it is not possible to obtain this information, we will adjust effect sizes using estimates from the literature of the ICC (e.g. Hedges & Hedberg, 2007), and assume equal cluster sizes. To calculate an average cluster size, we will divide the total sample size in a study by the number of clusters (typically the number of classrooms or schools).

Multiple intervention groups and multiple interventions per individual

Studies with multiple intervention groups with different individuals, and studies using multiple tests for the same intervention groups, will be included in the review. To avoid problems with dependence between effect sizes we will apply the robust variance estimation methods developed by Hedges, Tipton, & Johnson (2010). If we do not find enough studies in order for this method to consistently estimate the standard errors (Tanner-Smith & Tipton, 2014; Tipton, 2014) we will conduct a data synthesis where we use a synthetic effect size (the average) in order to avoid dependence between effect sizes. See below for more details about how we plan the data synthesis.

Studies including multiple interventions per individual may also be included, but only one intervention group (control or comparison group) will be coded and compared to the control or comparison group (intervention group) to avoid overlapping samples. We will choose the estimate from the intervention that we judge to have the least risk of bias.

Multiple studies using the same sample of data

In some cases, several studies may have used the same sample of data, e.g. studies using the same administrative data. We will review all such studies, but will only include in the meta analysis one estimate of the effect from each sample of data to avoid dependencies. The choice of which estimate to include will be based on our risk of bias assessment. We will choose the estimate from the study that we judge to have the least risk of bias.

Multiple time points

Outcomes will, if possible, be considered for the following intervals: •

Short-term effects (less than 3 months after the end of intervention).

We realize that 3 months is not a particularly long-term period. However, we expect having to use these definitions of short- and long-term effects based on the findings in Dietrichson et al. (2015a), where very few studies where found that reported outcome measurements more than 3 months after the end of intervention. Even fewer reported results after more than 6 months. If there are more studies reporting longer term effects found for this review, we will consider changing our definition of medium- to long term effects.

Data synthesis

The overall data synthesis in this review will be conducted where effect sizes are available. Studies that have been coded with a very high risk of bias (score of 5 in any item judged on a 5-point scale) will not be included in the data synthesis.

Random effects inverse variance weighted mean effect sizes will be used for all parts of the analysis and we will report 95% confidence intervals. The weighting function will be:

w i = 1 S E i 2 + τ 2

4

S E i 2

The analysis will be conducted in the following steps: Summary and descriptive statistics of the study-level contextual characteristics, methodological quality characteristics, group and subject level characteristics, as well as outcome characteristics will be used to describe the included studies. We will also include a correlation matrix with all moderators. Main effects analysis will be conducted first. Heterogeneity will be assessed with Chi-squared (Q) test, and the I-squared, and τ-squared statistics (Higgins, Thompson, Deeks, & Altaian, 2003).

If there is heterogeneity in effect sizes, we will perform a moderator analysis to attempt to identify the characteristics of study methods, interventions, and participants that are associated with smaller and larger effects on the various outcomes. We will use a mixed-model meta-regression to minimize the risk of misleading results due to correlated independent variables. We will start by pooling all effect sizes from studies with a treatment-control design (see below for a description of the analysis of comparison designs) and include the following types of moderators (variables in parentheses): •

Subject (math or reading test score)

The exact definition of the moderators may be subject to change during the data extraction process, but see Appendix B for preliminary version of the code book including more details on some of the moderators.

We will report 95% confidence intervals for regression parameters. To avoid problems with dependence between effect sizes we will apply robust standard errors (Hedges et al., 2010), using the Stata command robumeta. If there is significant heterogeneity also in the moderator analysis, this will warrant further examination of sub-groups. Sub-group examination could take the form of using interaction variables. However, we do not expect to find enough studies in order to run a meta-regression model where all relevant interactions are included. For example, interacting all instructional components and relevant combinations of these with all content domains while at the same time including other moderators would require a very large number of studies in order to not run into problems with degrees of freedom. On the other hand, it seems highly unlikely that the number of included studies will be so small that robust variance estimation becomes completely infeasible. The review of Dietrichson et al. (2015a) contains 25 studies of interventions targeting low SES students in middle school performed during 2000-2014. The target group for this review is broader and we will include studies of interventions further back in time. Therefore, we expect to include much more than 25 studies. We will use the simulation results reported in Tipton (2014) and Tanner-Smith & Tipton (2014) to assess how many moderators that can be included in each meta-regression.

If all moderators listed above cannot be included in the same regression due to limited degrees of freedom, we will proceed in two ways. First, we will exclude highly correlated variables, starting with moderators that have a higher correlation than 0.7, and then move down to 0.5, if necessary. Second, we will try factor analysis.

Subgroup analyses will be the next step. The primary objective of the review is to provide educational decision-makers with evidence of the effectiveness of interventions aimed to improve the results of educationally disadvantaged students. We will therefore focus the subgroup analysis on instructional methods and content domains. These are substantive features of interventions that for example teachers and school managers can affect, in contrast to other moderators (e.g. participant characteristics may be more difficult to affect for a school). The final categories of instructional methods and content domains will be developed during coding, but see section Interventions in practice for a description of what methods and domains that have been found in related reviews.

We will, if the number of studies allows it, use mixed-model meta-regressions and robust variance estimation in all sub-group analyses. The exact specification will depend on the outcome of the meta-regressions on the full sample of effect sizes. We will proceed in one of the following ways: 1) If there are enough studies on each instructional method/content domain so that an indicator for each component can be included in the meta-regressions using the full sample, we will get an indication of the comparative effectiveness of instructional methods/content domains from these regressions. That is, we can for example test whether one intervention component (or a combination of components) has a larger effect size than another, conditional on other moderators, with a t-test for the regression coefficients. 2) If it is not possible to test differences between instructional methods/content domains via t-tests of regression coefficients, we will evaluate the comparative effectiveness using similar methods as Wilson et al. (2011).

In 1) the subgroup analysis will aim to explain variation of effect sizes within the group of studies using the same instructional methods (if such variation exists). Alternatively, depending on the number of content domains in relation to the number of instructional methods, we will focus on explaining variation of effect sizes between content domains. In this type of sub-group analysis, we are not likely to be able to include all moderators in the same regression. We will then use a similar two-step procedure as described above: first exclude highly correlated variables, and then try factor analysis. If we do not have enough studies, and there is, as in Dietrichson et al. (2015a), small and insignificant differences between effect sizes in math and reading, we will pool studies using both math and reading interventions in the sub-group analysis of instructional methods (which are often the same across the two subjects) using a similar procedure with meta-regressions as described in the previous paragraphs.

In 2) meta-regressions of this kind are not possible. We will evaluate the comparative effectiveness using similar methods as Wilson et al. (2011); that is, adjust effect sizes using variables that do not share variance with the instructional methods/content domains. The procedure artificially makes every study equal on all the variables from the regression model. We will also show the unadjusted average effect sizes per instructional method/content domain for comparison.

For methodological quality, we will consider sensitivity analysis for each major component of the risk of bias tool. Statistical analyses will be conducted using Stata, as well as other software programs if needed.

Comparison designs

We will use comparison designs in the analysis only in cases where they may shed light on an issue, which could not be fully analysed using the sample of treatment-control studies. A concrete example may be that we, as in Dietrichson et al. (2015a), find relatively large but insignificant differences between tutoring interventions that are performed one-to-one and in small groups. Looking at comparison design studies that focus specifically on the issue of whether small-group tutoring can produce similar results as one-to-one tutoring, may then be useful to explore the variation in effect sizes. We will use meta-analytic techniques, including network analysis techniques (e.g. Higgins et al., 2012; Lumley, 2002; White, Barrett, Jackson & Higgins, 2012), to examine such questions if possible. If comparison design effect sizes cannot be pooled, study-level effects will be reported narratively.

Assessment of reporting bias

Reporting bias refers to both publication bias and selective reporting of outcome data and results. Bias from selective reporting of outcome data and results is one of the main items in the risk of bias tool.

We will use funnel plots for information about possible publication bias (Higgins & Green, 2011). However, asymmetric funnel plots are not necessarily caused by publication bias (and publication bias does not necessarily cause asymmetry in a funnel plot). If asymmetry is present, we will consider possible reasons for this. We will also use Egger's test, and test whether published studies have different effect sizes compared to unpublished studies.

Treatment of qualitative research

We do not plan to include qualitative research in the review.