PROTOCOL: The Effect of Linguistic Comprehension Training on Language and Reading Comprehension: A Systematic Review

The Problem

The ability to understand and express language in both its oral and written forms is a crucial aspect of human development. Developing linguistic comprehension skills is fundamental to all higher-level cognitive activities, learning, and sets the stage for reading development (McNamara & Magliano, 2009). Unfortunately, there are large differences in language comprehension skills among children (Biemiller & Slonim, 2001; Hart & Risley, 1995; Melby-Lervåg & Lervåg, 2014), and difficulties in reading comprehension are relatively prevalent among students across countries. In the U.S., 31% of the students in fourth grade and 24% of the students in eight grade performed below the basic level on the NAEP [National Assessment of Educational Progress] reading test in 2015 (National Center for Educational Statistics [NCES], 2015). The proportion of children reading below the basic level is even higher among children from families with low socioeconomic status (SES) and from minority race/ethnicity groups like black and Hispanic children. The situation of low-level reading skills among students is similar in North America and several Europeans countries (Organisation for Economic Co-operation and Development [OECD], 2010ab). Linguistic comprehension and reading comprehension skills are necessary for content-area learning in all subjects and thus are influential factors for academic success. Children who lack a strong foundation of linguistic and reading comprehension skills are more likely to experience academic difficulties and drop-out from school. Developing effective instructional practices is therefore of the utmost importance to the field of education.

This review aims to improve our understanding of intervention studies targeting two core constructs: linguistic comprehension and reading comprehension. Linguistic comprehension is defined as the process by which lexical (i.e., word) information, sentences and discourses are interpreted (Gough & Tunmer, 1986;). It refers to the ability to understand and express oral language, often assessed by tests of vocabulary or listening comprehension (Bornstein, Hahn, Putnick, & Suwalsky, 2014; Foorman, Herrera, Petscher, Mitchell, & Truckenmiller, 2015b; Klem et al., 2014, Melby-Lervåg & Lervåg, 2014). Vocabulary is a core component in linguistic comprehension. Vocabulary has typically been divided into either expressive and receptive vocabulary or depth and breadth vocabulary (Ouellette, 2006). However, several more recent studies using latent variables have shown that these are highly related constructs that are difficult to differentiate between (Bornstein, Hahn, Putnick, & Suwalsky, 2014; Klem et al., 2014). Although vocabulary is a core component in linguistic comprehension, skills such as syntax (the ability to understand and formulate sentences) and morphology (how words are formed), which build directly on vocabulary knowledge, are also often considered to be a part of a broader linguistic comprehension construct (Klem et al., 2014).

Reading comprehension can be defined as the active extraction and construction of meaning from all kinds of text (Snow, 2001). Linguistic comprehension is a well-known precursor to reading comprehension success that develops long before formal reading instruction begins (Snow, Burns & Griffin, 1998). A close relationship between linguistic comprehension skills and the development of reading comprehension has been demonstrated in several longitudinal studies (Lervåg & Aukrust, 2010; Muter, Hulme, Snowling & Stevenson, 2004; Storch & Whitehurst, 2002). According to the simple view of reading (SVR), linguistic comprehension is an important factor that underpins the development of reading comprehension beyond word-level reading (Gough & Tunmer, 1986). In later grades, when decoding skills are fully mastered and the contribution of decoding skills to reading comprehension has lessened, linguistic comprehension and reading comprehension are nearly isomorphic constructs (Muter, Hulme, Snowling, & Stevenson, 2004; Storch & Whitehurst, 2002). It is therefore worrying that large between-child differences in linguistic comprehension are observed from an early age (Hart & Risley, 1995). Hart and Risley (2003) found a large gap in vocabulary size between children from different socioeconomic groups as early as three years of age. Researchers also point to large differences in vocabulary knowledge between young second-language learners and first-language learners (Lervåg & Aukrust, 2010; Melby-Lervåg & Lervåg, 2014). Notably, between-child differences in the ability to learn words appear to be maintained throughout primary school (Biemiller & Boote, 2006; Lervåg & Aukrust, 2010; Melby-Lervåg et al., 2012). Due to these differences, intervention studies have aimed to boost development in linguistic comprehension skills in both preschool and school-age children and to examine the effects on linguistic comprehension and reading comprehension outcome measures (for reviews, see, e.g., Bus, van Ijzendorn & Pellegrini, 1995; Elleman, Lindo, Morphy, & Compton, 2009; Stahl & Fairbanks, 1986; Marulis & Neuman, 2010; Mol, Bus, de Jong, & Smeets, 2008).

As we will see, earlier meta-analyses have shown that transfer effects to global measures of linguistic comprehension and reading comprehension are typically much smaller and less impressive than effect sizes for custom measures (see e.g., Elleman et al., 2009; Marulis & Neuman, 2010). Whereas measures that refer to generalized language include test items that have not been explicitly trained in an intervention, customized tests are designed by researchers to show the effects of targeted training. Custom measures thus provide information on whether children have learned something that has been explicitly covered in an intervention (e.g., directly trained words in a training program). However, the ultimate goal of language-based interventions is to help children accelerate their further growth in linguistic comprehension and reading comprehension skills. If we are to narrow the gap between children with a low and a high vocabulary, we should focus on providing children with skills to continuously develop new knowledge of words. Unraveling the important factors that contribute to this generalization of knowledge thus becomes essential.

Despite the discrepancy in effects on global outcome measures vs. custom outcome measures, vocabulary training is recommended on a regular basis in schools. As noted by Coyne et al., (2010), researchers and practitioners often hold an implicit hypothesis that vocabulary instruction affords additional benefits beyond learning target words. However, the subsequent conclusions we are able to draw from these different assessment practices (custom vs. global outcome measures) will be of different practical value.

The primary aim of this review is to provide an overview of studies on interventions targeting linguistic comprehension and their effects on measures of generalized linguistic comprehension and/or reading comprehension. By strengthening our knowledge of this subject, we can potentially obtain insight into how related deficits can be ameliorated. This information is critical in making policy decisions about whether such programs are suitable for implementation in early childhood education and later schooling. Reviewing training studies may also provide a more refined understanding of the underlying causal mechanisms through which interventions are effective. This aspect is vital for providing a sound theoretical foundation for constructing better and more targeted intervention programs. In the following sections, we go into detail about how we plan to conduct this systematic review and how this review will contribute information not present in other reviews in the field.

The Intervention

Our review will focus on interventions that aim to improve linguistic comprehension and reading comprehension through linguistic comprehension tasks. In line with our definition of linguistic comprehension (see page 2), related tasks involve vocabulary (such as defining words and pointing at pictures of specific words), syntax (such as narrative skills, listening comprehension and building sentences), morphology (tasks related to the process of forming new words through diversion or composition), and other grammar tasks.

Several meta-analyses have summarized the literature on studies aiming to improve linguistic comprehension. Table 1 shows an overview of these meta-analyses. As shown in Table 1, when reviewing the literature, earlier studies that have attempted to improve linguistic comprehension through language-related tasks represent different study characteristics. Foci of instruction, outcome measures, participants (e.g., at risk, SES), implementation features (e.g., who is doing the training, dosage of intervention) and research designs are some factors that are likely to differ among studies and hence are important to incorporate as moderator variables in our review. Below we describe in more detail what constitutes a linguistic comprehension intervention program in regard to instructional approaches, measurement outcomes and research design. A more detailed description about studies to be included and excluded is given at page 22. Moreover, study characteristics will be coded and serve as potential moderators in our study. Information about this is presented at page 29 and in appendix II.

Research design

Table 1 shows that the intervention studies also differ with regard to other potentially important variations between studies. One core aspect in which they differ is related to study design. This variation involves three main issues: (1) whether the studies have a control group, (2) whether the potential control group is active or passive, and finally, (3) whether participants have been randomized across conditions. Moreover, they also differ in other important aspects, such as the amount and duration of training provided. Some of the previous meta-analyses include studies with as little as a one-hour intervention (see for instance Elleman et al., 2009). Table 1 further shows that the studies also differ in regard to the type of training (e.g., book reading, vocabulary intervention), location (i.e., home or educational settings), context (classrooms or small groups), the individuals providing the training (i.e., parents, teachers or researchers) and the participants (children with or without learning disorders and in different age groups). Thus, studies focusing on linguistic comprehension training cannot be understood as a homogeneous group of studies, and different intervention characteristics should therefore be taken into account when conducting a systematic review of this topic.

How the intervention may work

In this section, we outline the theoretical foundation for the review and develop a model for the relations that are to be tested in the review.

Model of the relations between intervention, moderator variables and outcomes

Figure 1 illustrates the causal mechanisms behind the intervention. As can be observed, linguistic comprehension is hypothesized to be a mediator that affects the strength of the effects on reading comprehension skills. However, we also predict important direct effects on linguistic comprehension. Moderators in the model are hypothesized to have an impact on the effects on both linguistic comprehension and reading comprehension. Notably, the indicators that are considered representations for the latent constructs in the model (and coded in the systematic review) are clearly defined in the next section.

OPEN IN VIEWER

1

Model of the relations studied in the review

Why it is Important to do the Review

Table 1 shows that our literature search found 12 previous reviews on this topic. Thus, it may be asked why there is a need to perform an additional review. As we demonstrate in this section, although several reviews have been published on this matter, they do not fully test the model that we aim to examine in this systematic review. Additionally, several of the previous meta-analyses are now outdated; many new studies are not included. The incorporation of these new studies makes our review substantially different from and a valuable extension of recent reviews.

As shown in Table 1, earlier meta-analyses have included studies that use both customized tests for the targeted training and standardized tests in their examination of training effects. However, in many of the previous meta-analyses, a mean effect size was calculated to combine these two test types (see Table 1, Mol, Bus, & de Jong, 2009; Swanson et al. 2011). This procedure yields a biased result for the effects of linguistic comprehension training. In contrast, the planned review will exclusively examine studies that report standardized measures in addition to measures of the words that are trained. For the overall mean effect size, the effects on standardized tests and custom measures are calculated separately. Importantly, we plan to examine whether the effects on standardized measures are mediated by the effects on custom measures. This approach enables the review to make a unique contribution to the study of the effectiveness of linguistic comprehension intervention in educational settings.

Several meta-analyses on the topic have exclusively examined the value of shared book reading (e.g., Blok, 1999; Bus, van Ijzendoorn, & Pellegrini, 1995; Mol, Bus, de Jong, & Smeets, 2008), whereas others have included several types of vocabulary interventions in addition to print-based training (Elleman et al., 2009; Marulis & Neuman, 2010). Similar to Marulis and Neuman (2010) and Elleman et al. (2009), we want to include training studies that focus on both shared book reading and other types of vocabulary instruction. We will include studies with children older than those in Marulis and Neuman (2010), who examined children from birth to age nine. We will also include studies that contain a broader view of oral language training (e.g., training focusing on listening comprehension, narrative skills, and morphology/grammatical skills). Such studies have been published more recently and have not been included in any of the previous meta-analyses (e.g., Clarke et al. 2010, Fricke et al., 2013).

Another reason for conducting this review is to examine the question of how dosage (duration and amount of training) of training is related to intervention effects on global language comprehension skills. Earlier meta-analyses do not give an answer to this question. The previous work by Elleman et al. (2009) and Marulis & Neuman (2010) found that longer duration of treatment is not related to higher effect sizes in comprehension. However, because Elleman et al. (2009) and Marulis & Neuman (2010) include studies with both global and custom outcome measures in their analyses, this finding is not applicable to the question of how dosage of training is related to increasing global comprehension skills (i.e., outcome measures of generalized language). As Elleman et al. (2009) note from their study, studies using custom measures of vocabulary were short in duration (more than half of the studies were conducted in less than 10 hours) and studies using a standardized measure, in general, had longer interventions. The finding that duration is not an important factor to increase vocabulary skills is not surprising considering the included studies and method of analysis in these reviews. In contrast, we can hypothesize that the picture will look different when it comes to interventions designed to increase global language skills.

Regarding settings, the above-mentioned meta-analyses vary in their inclusion criteria. Bus et al. (1995) and Mol et al. (2008) studied book reading in parent-child settings and excluded interventions implemented in educational settings. Blok (1999) and Elleman et al. (2009) included only training studies in educational settings, whereas Marulis and Neuman (2010) included training studies implemented in both home and educational settings. Our aim is to focus on language training conducted exclusively in educational settings because these studies have the most relevance for educational policy and practice. Thus, we want to exclude interventions implemented by parents or in the child's home environment. An additional reason for this is that this review will focus on how linguistic comprehension and reading comprehension can be improved in an educational setting.

Also, this review will be the first to expand the current literature in incorporating training studies from both an early age in preschool and school-age children. We include studies conducted in preschool and later educational settings up to the end of secondary school. Notably, the National Early Literacy Panel (2008) studied shared-reading interventions in children aged zero to five, and no studies examined the impact of intervention on reading as an outcome variable. Similarly, Marulis and Neuman (2010) targeted only the very early years of vocabulary development (birth through age 6) and did not include measures of reading comprehension. Elleman et al. (2009) examined the impact of vocabulary instruction on reading comprehension in school-age children where the majority of the studies include instruction conducted in grades 3 to 5.

Further, many of the previous meta-analyses included studies without an appropriate control group, e.g., within-subjects designs (see for instance Table 1, Elleman et al. 2009) or comparisons between pre-existing groups (Table 1, Marulis & Neuman, 2010). Our review will only include information from randomized controlled trials and quasi-experiments with a control group and measures of baseline differences. We will also focus on the type of control group that is used in the study (more details about this is given in the coding manual, appendix II). Several studies have shown that this can be an important factor in explaining differences between studies (see Boot et al. 2013). Further, in contrast with the majority of previous reviews (see Table 1), we will also code measures of follow-up effects if this is reported because the practical value of such interventions also depends on the extent to which intervention effects are lasting. Altogether, this emphasis on methodology in our review generates the best data for drawing conclusions on the effectiveness of intervention programs.

In summary, our planned review will take a different approach in summarizing linguistic comprehension training studies than earlier reviews on the topic. The primary interest and need for this review are linked to the question of whether linguistic comprehension training effects can be transferred to standardized measures. To obtain more knowledge on this point, this review will take a different approach to the type of training studies that will be included compared to prior reviews (e.g., types of research designs and outcome measures). The planned review will expand the current literature by including recent research from the past several years. For instance, Elleman et al. (2009) included studies from 1950 to 2006, and Marulis and Neuman (2010) conducted their search through September 2008. In addition, we will conduct a thorough search for grey literature in collaboration with an information retrieval expert. Altogether, this approach will make our study a valuable addition to previous reviews. Based on this most current knowledge base, the review may be able to answer important policy questions related to the extent to which children's global linguistic comprehension and reading comprehension skills can be altered through interventions and, if so, how interventions that promote global linguistic comprehension and reading comprehension can be best constructed and implemented.

Inclusion and exclusion criteria for the studies in the systematic review

Search Strategy

The search for relevant literature will include studies published in English since 1986. The search will use multiple sources of information and will be supervised and conducted in collaboration with an information retrieval specialist at the Library of Humanities and Social Sciences, University of Oslo. Details of the search strategy are included in Appendix I.

Procedures for screening and inclusion of studies

Figure 2 below shows the procedure for screening studies to be included in the review.

Description of methods used in primary research

Randomized controlled trials and quasi-experimental studies with pre-post designs and control/comparison groups will be included in the review.

The use of pretests in the studies' designs is required to determine whether groups are equivalent and comparable prior to a given intervention program. Posttest measurements in the studies enable us to compute group differences and changes over time. Most studies include a post-test measure conducted immediately after the intervention program has ended. Some studies may also include data from follow-up measurements at later points in time, which we will include in the review.

Criteria for the determination of independent findings

Violating the assumption of independence by computing an overall effect size based on information from the same sample more than once can lead to biased estimates (Borenstein et al. 2007). Thus, several considerations will be made before the studies were coded.

Multiple reports of the same study. It is possible to come across multiple reports from one single study. In such cases the coders will make sure to have the total number of reports before the coding process starts. All the different reports may contribute with additional information to the coding manual. The review authors will choose to extract information from the report that best fit the information needed for each item in the coding manual. Multiple publications within our sample of studies can be detected by finding characteristics such as identical sample sizes, authors, intervention programs or outcome reports. Because multiple publications can lead to an incorrect weighting of study results, authors will be contacted if there are uncertainties regarding the multiple publication of original research.

Multiple studies in single reports. Several studies may be described in one single report. In such cases, each study will be coded separately.

Multiple comparison groups and multiple interventions. It is likely to come across studies that make use of several control groups. Only the intervention and control groups that meet the eligibility criteria will be included. Also, some of the studies are likely compare the same control to different experimental groups and are included in the same analysis of a mean effect size for treated and untreated controls. Since correlations between the multiple comparisons and their outcomes are likely to be rarely reported in the original studies, we will include these studies in the analyses assuming zero correlation between the outcomes.

Multiple outcomes. When a study has multiple indicators for the same construct (for instance more than one global measure of reading comprehension) the mean of the indicators will be used.

Multiple time-point. When a study has multiple time-points, we will use pretest, immediate posttest and the follow up measure that is closes to 6 months after the intervention.

Details of study coding categories

Differential effects from the different studies may be influenced by systematic differences between the studies. Thus, the coding and examination of moderator variables will account for differences between the studies. Studies will be coded on variables related to:

Contextual information (e.g., year, type of publication)

Examining the Strength of Evidence using GRADE systems are available for rating research-based evidence (Sanderson, Tatt, Higgins, 2007). In this review, to assess the strength of the evidence, we will adapt the Grading of Recommendations, Assessment, Development and Evaluation system (GRADE; Guyatt et al., 2008; 2011). The GRADE system rates the quality of evidence and is used widely by, for instance, Cochrane and the World Health Organization (Guyatt et al., 2008; 2011). The quality reflects the confidence in the results based on the study limitations and risk of bias. The Cochrane risk of bias tool will be used to assess risk of bias (Higgins et al., 2011). There are three categories: low risk of bias, unclear risk of bias and high risk of bias. Each study will be rated in the “Risk of bias table” (see appendix 2) where judgement of categories (low risk of bias, unclear risk of bias and high risk of bias) will be made according to the domains; sequence generation, allocation concealment, blinding, incomplete outcome data; selective outcome reporting; other sources of bias. Dual assessment of risk of bias will be conducted. In addition to the risk of bias tool, more information related to methodological information will be coded to assess information about study quality (see appendix II).

Instructional methods (type of training; definitional-contextual scale; type of exposure; level of discussion; type of text involved in the training; depth of processing). These characteristics are included in order to examine why some interventions may be more effective than others. Some of the categories are adapted from Stahl & Fairbanks (1986) and was also used in the study by Elleman et al., 2009).

More about the coding procedures is given in Appendix II.

Statistical procedures and conventions

The analyses will be conducted using mainly the “Comprehensive meta-analysis” program (Borenstein, Hedges, Higgins, & Rothstein, 2005). We will calculate effect sizes by dividing the differences in gain between pre- and post-test in the treatment and the control group by the pooled standard deviation for each group at pretest; this method of effect size calculation for pretest-posttest control group designs is recommended (Morris, 2008). Thus, when the effect size is positive, the group receiving training has made greater pretest-posttest gains than the control group. We will adjust the effect size for small samples using Hedges g (Hedges & Olkin, 1985). Effect sizes for follow-up tests will be calculated in an analogous way (pretest to follow-up).

The mean effect sizes will be calculated by a weighted average of individual effect sizes using a random-effects model. Outliers larger than 3 SDs from the mean will be excluded. Notably, since we examine effects from interventions that are rather different when it comes to intervention content and age group they are targeted towards, depending on the number of studies it will be preferably to do separate analyses of the main categories of interventions (i.e. shared book reading, vocabulary training, general language training).

For moderator analyses, we would prefer to the whole model indicated in Figure 1 using a structural equation procedure (MASEM). However, given the potential number of studies and issues with missing data (and that very few experiments tend to report correlations), this is not likely to be realistic because of problems associated with the number of parameters versus the number of studies. However, we will use meta-regression procedures to test aspects of the models in different analyses. Notably, the moderators in Model 1 are restricted to the primary issues that, based on prior studies, have demonstrated clear importance. Increasing the number of moderators is bound to result in Type 2 errors in the analysis that we plan to perform. Thus, we plan to test only the moderators for which there are clear theoretical motivations. Rather than using MASEM it is likely that we will use method of moments meta-regression for continuous variables (e.g. age, training duration etc). To examine whether effects on reading comprehension are mediated through linguistic comprehension gains, we will set up mediation models using meta-regression. For categorical moderator variables, studies will be separated into subsets based on the categories in the moderator variables (e.g., experiments vs. quasi experiments). A Q-test will be used to examine whether the effect sizes differed between subsets. When there are fewer than four studies in a subset (k < 4), this analysis will not be conducted. The overlap between confidence intervals will be used to examine the size of the difference between subsets of studies.

Publication bias refers to the notion that a mean effect size can be upwardly biased because only studies with large or significant effects get published (i.e., file-drawer problem with entire studies), or that authors only report data on variables that show effects (often referred to as p-hacking, or the file-drawer problem for parts of studies; see Simmons, Nelson, & Simonsohn, 2011). In line with recommendations for meta-analyses, we will make special efforts to retrieve studies from the grey literature and used this as a moderator when possible (Higgins & Green, 2011). To estimate the impact from publication bias statistically, commonly funnel plots have been used in combination with a trim-and-fill analysis. However, there are several problems with the funnel plot/trim-and-fill method (Lau, Ioannidis, Terring, Schmid, & Olkin, 2006). P-curve is a recently developed method that deals with the weaknesses in the funnel plot/trim- and-fill analysis (Simonsohn, Nelson & Simmons, 2014). Ap-curve plots the distribution of statistically significant p values (p < 0.05) in published studies, and the shape of the p-curve is a function only of the effect size and sample size, when the power level is taken into account. If there are true effects, one expects the distribution of published p-values to be right-skewed with more low (.01s) than high (.04s) p-values. However, if a set of studies is affected by publication bias (because researchers discard entire studies or discard analyses or parts of studies), the p-curve becomes left-skewed or flat. Such a form of p-curve is said to provide “no evidential value” (i.e., no support for an appreciable effect size). For studies that have published significant findings we will use p -curve to examine bias in these findings.

When coding articles, it is expected that there will be numerous instances of missing data. If data is critical to calculate an effect size, articles with missing data will be excluded if authors do not respond to an email request to provide the data. In cases where an effect size cannot be computed on one outcome but data were missing on other outcomes or moderator variables, the study will be included in all the analyses for which sufficient data were provided.

Treatment of qualitative research

Qualitative research will not be included in the review.