PROTOCOL: The Direct and Indirect Effects of School-Based Executive Function Interventions on Children and Adolescents’ Executive Function,Academic,Social-Emotional,and Behavioral Outcomes: A Systematic Review

TITLE OF THE REVIEW

The Direct and Indirect Effects of School-Based Executive Function Interventions on Children and Adolescents’ Executive Function, Academic, Social-Emotional, and Behavioral Outcomes: A Systematic Review

BACKGROUND

EXISTING REVIEWS

The rising public awareness of the importance of executive function has led to a wealth of interventions designed to promote and enhance executive function skills in children and adolescents. Commonly used executive function interventions comprise two categories: pharmacological (e.g., stimulants used to treat ADHD patients) and non-pharmacological (e.g., computerized training, cognitive behavior therapy, and school-based interventions). As is the case for pharmacological interventions, researchers have conducted a number of systematic reviews or meta-analyses to investigate how effective non-pharmacological interventions are and whether their effectiveness was supported by reliable and convincing evidence. A big portion of such reviews are conducted on clinical samples or laboratory-based settings on non-school-based interventions such as dietary (e.g., free fatty acid supplementation) and computerized training programs (e.g., computerized working memory training programs) (e.g., Karch, Albers, Renner, Lichtenauer, & von Kries, 2013; Melby-Lervåg & Hulme, 2013; Sonuga-Barke et al., 2013; Wayne et al., 2014). Such reviews may not be particularly useful and informative for stakeholders of education arena.

Some researchers have reviewed the effects of school-based executive function interventions on normal children and adolescents, particularly some direct outcomes relevant to executive function, such as general EF skills and/or specific skills in inhibition, working memory, and cognitive flexibility. Most often, these researchers chose to limit their scope of reviews to one particular form of EF interventions. For example, Smith et al.'s (2010) meta-analysis synthesized randomized controlled trials on the impact of aerobic exercise on attention and processing speed, executive function, and memory performance of adults over 18 years of age (also see Maynard, Solis, & Miller, 2014; Randolph, Rosenstein, & Michaels, 2014).

Additionally, there exists a number of reviews focusing on the effects of school- or home-based executive function interventions. Most such reviews examine indirect rather than direct outcomes of executive function interventions, such as academic achievement, social-emotional development, or behavioral performance. For example, Jacob and Parkinson (2015) conducted a systematic review of the association between executive function and student academic achievement in reading and math. Randolph et al. (2014) developed a Campbell Collaboration (C2) systematic review protocol on the effects of Montessori education on academic and behavioral improvements among 5-12 years old elementary students. Similarly, Maynard et al.(2014) was another C2 review protocol on mindfulness-based interventions for improving academic achievement, behavioral and socio-emotional functioning of primary and secondary students. Currently, Baron, Evangelou, Malmberg, and Melendez-Torres (2016) are conducting a systematic review concerning the effectiveness of Tools of the Mind on young children's self-regulation, particularly in comparison to other similar school curricula or programs.

The What Works Clearinghouse (WWC) have also reviewed research on the effectiveness of some executive function interventions. For example, the WWC (2008) reviewed research evidence for Tools of the Mind, an early childhood curriculum for preschool and kindergarten children. They found that one study (Barnett et al., 2008) met their evidence standards, three studies did not meet their evidence standards, and seven other studies did not meet their eligibility screens. Similarly, the WWC (2011) reviewed the research evidence of the Incredible Years program (see Webster-Stratton, 2011; Webster-Stratton, Reid, & Hammond, 2004). It is noteworthy that the WWC reviews are different from Campbell systematic reviews. Specifically, the WWC reviews aim to identify the best available evidence on an educational intervention, rather than the general effectiveness of an intervention or the potential moderators of the effects.

There also exist a number of reviews of studies that do not focus on executive functions but broadly investigate the effects of school- or home-based social, emotional, or behavioral interventions on normal or disadvantaged young children and adolescents. Such interventions include self-monitoring interventions (e.g., Thompson, Maynard, Bowen, & Pelts, 2013), group-based parent trainings (Barlow & Parsons, 2005), social competence interventions (e.g., Kennedy & Pigott, 2012), instructional strategies (e.g., Spivak, Lipsey, Farran, & Polanin, 2013), school-based universal interventions (e.g., Durlak et al., 2011), and after-school programs (e.g., Durlak, Weissberg, & Pachan, 2010). These reviews typically concentrate on outcomes such as reducing challenging behaviors, developing prosocial behaviors, social competence and performance, improving academic achievement in children, adolescents, or young adults.

Taken together, our survey of existing and ongoing executive function interventions reviews suggests there are no systematic reviews that have all of the following three features: (a) integration of non-pharmacological executive function intervention research from fields like neuroscience, neuropsychology, and psychology with research in education, with a primary focus on serving and informing stakeholders in education; (b) inclusion of multiple types of school-based interventions for typically developing children and adolescents, such as school curricula/educational programs or mind-body interventions; and (c) examination of both direct (e.g., general or specific executive function skills) and indirect intervention outcomes (e.g., academic achievement, social-emotional development, and behavioral performance). We thus propose to conduct a meta-analysis that will have all these three features.

THE CURRENT META-ANALYSIS

METHODOLOGY

Population

This meta-analysis will include studies with populations of typically developing children and adolescents from ages 3 to 18. There are three reasons for selecting early childhood through adolescence as the developmental period of focus in this review. First, this age range is the key developmental period during which foundational executive functions, such as inhibition, working memory, and cognitive flexibility, rapidly progress (Best & Miller, 2010; Best et al., 2009). Second, evidence supports that early childhood and adolescence are significant periodsduring which to intervene for executive function development through intentional teaching and support due to brain plasticity (Cramer et al., 2011). Lastly, selecting this age range corresponds to the main objective of this review, which is to provide evidence-based information regarding the effects of executive function interventions that can guide educational practices for children and adolescents to enhance outcomes. There will be no exclusion criteria based on study participants’ demographic characteristics. Eligible studies may have been conducted on participants of any country but must be reported in English. Studies that have no full-texts available in English will not be included as translating such studies into English will greatly extend the time needed to complete the current meta-analysis.

In other words, this meta-analysis will exclude studies conducted on primarily special populations that were clinically or medically diagnosed with biological, neurological, mental, behavioral, or learning disabilities. Examples of such special populations include children and adolescents with major cognitive deficits or brain injury, diagnosed with ADHD or attention deficit disorder (ADD), autism spectrum, oppositional defiant disorder (ODD), conduct disorder (CD), or disruptive behavior disorders (DBD), or other severe disorders. Many existing systematic reviews or meta-analyses have been conducted to synthesize the effects of EF interventions on such special populations (e.g., Cortese et al., 2015).

Study design

Eligible quantitative studies need to employ strong empirical experimental research designs, particularly randomized controlled trials (RCTs). Randomized controlled trials (RCTs) include, but are not limited to, parallel group trials, cross-over trials, cluster-randomized trials, and factorial trials. Studies need to employ appropriate comparison groups, which ought to resemble the main characteristics of the intervention groups in terms of the baseline measures, age, grade levels, or other demographic factors, or these features can be statistically controlled to discount the prior differences.

Time and language

All eligible studies have to be conducted during the period from January 1985, when most of the school-based EF interventions began to be prevalent, to the time of literature search conclusion (approximately February 28^th, 2017) and their full texts have to be available in English.

Search Strategies

We will use the following six strategies to identify published or unpublished eligible studies:

Search in electronic databases

1)

Academic Search Premier

Web search using search engines

1)

Google

Grey literature search

1)

The Society for Research on Educational Effectiveness

Track bibliographies of previous literature reviews and retrieved studies

We will screen the reference list of prior reviews and related meta-analyses conducted between 2000 and 2016 for additional relevant studies. We have identified 11 relevant previous meta-analyses or reviews that are useful for us to identify eligible studies (see Appendix). We will also examine the reference of the retrieved primary studies for potentially eligible studies. This procedure will be carried out not only in the search stage but the study coding process.

Handsearching

We will conduct handsearching of the Table of Contents of at least 3 journals which published most of the identified eligible studies, including Developmental Science, Journal of Consulting and Clinical Psychology, and Society for Research on Educational Effectiveness.

Contact leading authors

We will contact authors who were the first authors of two eligible studies to request unpublished or in-pressstudies.

Search Keywords and Terms

We have conducteda pilot review of a set of primary studies and existing reviews to identify relevant free text terms/keywords. We have also consulted database-specific thesauri to locate relevant standardized subject terms (i.e., controlled vocabulary). We will use combinations of terms/keywords related to the intervention, study design/focus, targeted population, and outcomes to conduct the search. Database-specific strategies will be explored for each database, including the use of truncation (such as “”), wildcard (e.g., *), Boolean operators (AND, OR and NOT), and limiting commands to ensure search sensitivity and precision. Below are examples of search term combinations we plan to use: 1)

Intervention: “executive funct*” OR “executive control*”OR “executive systems*”, “executive skills*”OR “inhibitory control*”OR “cognitive flexibility*” OR “task-switching*”OR “working memory*”OR “attention*”OR “supervisory attentional system*”OR “cognitive control*”OR “everyday cognitive skills*”OR “effortful control*”OR “neurocognitive functioning*”OR “conscious control*”OR “behavioral regulation*”OR “emotion regulation*” OR “regulatory functions*” OR “mindful*”
AND

In addition, we will conduct searches with the names of major EF interventions, such as: •

“Tools of the Mind”

Procedures for Determining Eligible Studies

Two trained research assistants will independently conduct the initial search following the search strategies above. They will screen the titles and abstracts of all search hits and collect the studies that are judged to be eligible. These studies are called “first-stage-eligible studies”. They then will cross-screen each other's list of “first-stage-eligible studies” and meet every other week to examine the consistency or discrepancies between their lists of studies. After each two-week period, they will meet with the leader author to go through the outcomes of their cross-screening meetings. Studies determined to be eligible after these processes are called “second-stage-eligible studies”.

At the time of this revision, we have preliminarily identified 45 randomized controlled trials (RCTs) that evaluated the effects of EF interventions and met all inclusion criteria of this meta-analysis. Table 2 presents these identified 45 RCTs, their primary outcome(s), and key assessments of the outcome(s). Study search and screening will be completed by approximately February 2017. The full text of all second-stage-eligible studies will be obtained and enter the coding stage. Only studies that fully meet all inclusion criteria on the basis of their full texts will be determined eligible and included in the analyses.

Description of Exemplary Eligible Studies

This meta-analysis will include studies that are randomized controlled trials (RCTs) that compare outcomes for an intervention group to those for a control or comparison condition. Interventions were implemented with students in P-12 educational settings, either in a class, school, or after school program. In some studies, the inventions were delivered to children/adolescents, but in others, the interventions were delivered to teachers, school psychologists, or relevant school personnel. At least one key outcome had to be relevant to one of the two primary outcomes that this meta-analysis is interested. Most potentially eligible studies include both pre-test and post-test measures for both intervention and comparison groups. Post-test measurements generally occur at the end of the intervention. Below are summaries of two representative eligible studies:

Study Coding

Study coding will be conducted using a coding form (see Table 4 for the preliminary coding protocol). We will conduct a pilot coding of approximately 10 studies to test and fine-tune the preliminary coding form before proceeding to code all eligible studies. This coding protocol covers the major characteristics of eligible primary studies, such as (a) general features of the primary studies (e.g., the forms of EF intervention, context, nature, implementation, and the length of intervention), (b) the methodological features of the studies (e.g., research designs, participants, sample sizes, and outcome measures), and (c) effect size information (e.g., the statistical information needed to calculate effect sizes).

In the coding process, we will record the data in an Excel spreadsheet for the convenience of data management. To ensure the reliability of coding procedures and decisions, the leader author and a research assistant will independently code all the included studies and will compare the coding for each study, compute an inter-rater agreement, and resolve any coding discrepancies. The first author will meet regularly with the second and third authors to discuss difficult coding issues.

In cases of studies with missing data, we will first attempt to search and examine all relevant documents that might contain helpful supplemental information. Examples of such documents might be multiple versions of a study that were published in outlets such as research journals, books, or conference proceedings. We will also contact the lead author of the studies to request assistance if needed.

Effect Sizes

We will use standardized mean difference effect sizes (i.e., Hedges’ g) for outcomes on continuous measures and odds ratios (OR) for outcomes presented as dichotomous variables. Hedges’ g can reduce the bias that may arise when the sample size is small (i.e., n < 40; Glass et al., 1981; Hedges, 1981). The What Works Clearinghouse (WWC) (2013) adopted Hedges’ g as the default effect size measure for continuous outcomes in the WWC reviews. We will take caution in examining the nature of these outcomes to ensure that a positive ggenerallyindicates that the treatment group outperforms their counterparts in the comparison condition. We will use two computational tools for effect size calculation. The first one is the Comprehensive Meta-Analysis (CMA) software (Borenstein, Hedges, Higgins, & Rothstein, 2006). The CMA can calculate effect sizes from more than 100 types of data formats on the basis of a wide variety of statistics (e.g., means, standard deviations, and p-values) and types of data (e.g., binary, continuous, and correlational data). Another tool is David Wilson's practical effect size calculator, which can be accessed online free of charge through http://www.campbellcollaboration.org/resources/effect_size_input.php.

Adjusted and Unadjusted Hedges’ g. Two types of Hedges’ g will be extracted based on the information available in the studies. If a study only reported post-test outcomes, we will extract unadjusted Hedges’ g which does not take into account the confounding impact of other variables on the outcomes. If a study provided outcomes that either adjusted or controlled for the influence of other variables (e.g., pre-test scores) or reported information that will allow us to do so, we will extract adjusted effect sizes. In some cases, adjusted effect sizes will be based on means and standard deviations of gain scores (i.e., post-test minus pre-test scores), whereas in other cases they might be based on covariance-adjusted means and standard deviations. Extracting two types of Hedges’ g will allow us to make the best use of study information and examine whether the effectiveness of school-based EF intervention differs depending on how it is estimated.

Effect sizes by comparison conditions. We will extract effect sizes corresponding to each comparison condition (e.g., regular curriculum, waitlist control, no treatment, or other alternative activities) to which the school-based EF intervention is compared within each study. For example, if a study compared the effects of Tools of the Mind curriculum with both regular curriculum and no treatment control, two separate effect sizes will be computed respectively with this study.

Conversion among different effect sizes. After all effect sizes are calculated, we will choose one type of effect sizes that appears most frequently as the primary effect size index (most likely Hedges’ g), and convert other types of effect sizes (e.g., odds ratios) into the primary effect sizes prior to computing the mean effect sizes (Borentein et al., 2009). All effect sizes that entering the analysis will be Hedges’. The Comprehensive Meta-Analysis (CMA) can simultaneously compute and display various types of effect size (e.g., Hedges’ g, odds ratios, log odds ratios, correlation coefficients, and Fisher's z, etc.) so this is achievable.

Data-Analysis

Compute mean effect sizes. For each meta-analysis, we will conduct the analysis with a random-effects model, which assumes that the treatment has more than one true effect size, and the study effect sizes are a random sample of a population of effect parameters. This model assumes that, in addition to sampling errors, some other variable, such as study setting, the length of the treatment, and the specific features of research designs, may be responsible for the variation in observed effect sizes (Schmidt & Hunter, 1977). Effect sizes from each study will be weighted with the Hedges–Vevea method which weights the study level effect sizes by inverse variances (i.e., dividing the value of one by the variance or standard error squared). It is worth noting that using inverse variance weights in a random-effects model includes an estimate of between-study variances along with within-study variances. We will report the mean effect sizes with their 95% confidence intervals, and p-values of tests of the null hypotheses.

Ensure effect size independence. In each analysis, ensuring effect size independence is a prerequisite. Effect size dependency may arise from many situations. We will use different methods to ensure effect size independence under different circumstances. We briefly describe four strategies and their related circumstances below.

First, we will use an independent sample as the unit of analysis (Cooper, 2010) to integrate effect sizes . With this approach, each independent sample contributes only one effect size to a mean effect size. If one research report contains two or more separate studies based on different samples, we will count each separate study (associated with each respective independent sample) as a unit. As a result, the number of independent samples might be greater than the number of eligible study reports in the end. In cases in which more than one article reported study findings that were all based on the same sample, we will choose the study (article) of the sample with best design and/or psychometric quality to represent all relevant studies (reports).

Second, we will conduct separate meta-analysis corresponding to each category of conditions to which the school-based EF intervention was compared. For example, three separate meta-analyses will be conducted on three groups of studies (i.e., independent samples) in which EF interventions were compared to traditional school curriculum, no treatment control, or other alternative activities, respectively. When effect sizes are grouped by each type of comparison condition, all effect sizes associated with one type of comparison will be independent from one another because each sample contributed one effect to the estimation of that comparison.

Third, most commonly, effect size dependence arises when a single study produces more than one effect size due to multiple outcomes. These effect sizes are based on the same group of subjects and therefore are not independent. In such a situation, we will analyze the effect sizes in two stages. In the first stage, we will compute an average effect size across all available outcome measures in each study. This average effect size will represent the effect size estimate of this study. For example, if a study measured a group of participants with three different measures, the average effect sizes from these three measures will be the effect size contributing to the overall average effect size across studies.

In the second stage, we will conduct analyses using a shifting unit of analysis approach (Cooper, 2010). The preparation for using this approach will start in the effect size calculation in the coding stage. Particularly in the coding stage, one effect size will be calculated for each major outcome measure. For example, if astudy (one independent sample) used both standardized test scores and course grades to measure students'learning, two separate effect sizes will be calculated. With the shifting unit of analysis approach, this study will contribute one effect size associated with the standardized test scores to a meta-analysis that focuses on integrating effect sizes from standardized outcome measures. Similarly, this study will contribute one effect size corresponding to the course grade to another meta-analysis that focuses on integrating effect sizes from non-standardized outcome measures, such as course grades. In sum, with shifting unit of analysis approach, although the two effect sizes are associated with the same sample, they are independent from one another when they each contribute to a different meta-analysis.

We will also utilize several multilevel meta-analysis software packages such as MLwiN or Metafor in R to help handle effect size dependency issues arising from the same study.

Heterogeneity analysis. We will conduct heterogeneity analysis to statistically assess the variation in study level effect sizes. Key statistics will include the value of total variance (Q statistic), the results of chi-square-like test based on the Q statistic (p-value), the between-study variance (i.e., the true heterogeneity) (T² ), and the ratio of between-study variance to total variance (I ²). I-squared (I ²) ranges from 0 to 100 (%). The Q statistic and I ² are algebraically related (Higgins & Thompson, 2002).

Moderator analysis. We will conduct moderator analysis (also called testing for moderators) to explore possible sources of heterogeneity in the effect sizes. In particular, based on prior theory and research literature, it explores whether the differences inkey characteristics significantly change the magnitudes and/or direction of the mean effect sizes. Essentially, moderator analysis examines the degree to which the relation between the independent variable (i.e., study characteristics) and the dependent variable (i.e., the mean effect size) changes by study-level or person-level characteristics (Hedges & Pigott, 2004).

A tentative list of potential moderators that this meta-analysis will test include:age, gender, grade level, types of interventions, major components of interventions, the duration of interventions, the fidelity of intervention implementations, research design (i.e., comparing an complete RCT to clustering, matching, or other non-standard design features of RCTs) study methodological quality, the type of information sources (i.e., self, teacher, parent, etc), and rewards (a factor that some researchers believe it has some interesting relationship with EF) will be examined for their influence on the mean effect sizes.

For categorical variables, we will conduct moderator analysis with a method analogous to the analysis of variance (ANOVA) in the data-analysis of a primary study (Borenstein et al., 2009; Huizenga, Visser, & Dolan, 2010), also called subgroup analysis (Borenstein et al., 2009). Q _b and p-values are two key statistics in the ANOVA analog method. A Q_b denotes the total between-group variance associated with the sub-categories of a given moderator variable. A p-value denotes the results of a significance test for the mean difference between or among the subgroups. If there are at least 10 eligible studies for a subgroup, we will conduct meta-regression with which we may explore the relationship between some continuous variables (e.g., age) and the mean effect sizes.

To minimize the chance of Type I errors in moderator analyses – the increased chance of finding at least one significant result as more tests are added even if that is not true in reality, we will avoid conducting moderator analysis on too many variables, especially when the number of included studies in the meta-analysis is limited. Meanwhile, we will consider implement adjustment for multiplicity (i.e., adjustment of alpha level), such as using a Bonferroni (more conservative) correction approach in the analyses.

Assess Publication Bias and Risk of Bias

We will first visually assess publication bias through a funnel plot. Further, we will conduct the trim and fill analysis (Duval, 2005; Duvall & Tweedie, 2000), to unearth the existence of publication bias, to evaluate the sensitivity of results to possible publication bias, and to adjust results when there is a suspicion of publication bias. In addition, we will also conduct sensitivity analysis to assess the robustness of the results.

We will employ the Cochrane Collaboration's tool for assessing risk of bias (Higgins et al., 2011) given that all included studies will be RCTs. The outcomes of assessing risk of bias maybe candidates for moderator analyses if they fit.

Interpret Meta-Analysis Results

We will use the “meta-analytic thinking” approach (Thompson, 2006) to interpret the meta-analysis results. This approach generally involves taking into consideration multiple contextual factors in interpreting results. First, we will consider multiple factors in interpreting results, including the size of the effect, statistical significance, confidence intervals, and the number of studies providing the evidence. This approach is similar to the What Works Clearinghouse's (WWC) recent guidelines for determining the evidence rating for an intervention in WWC reviews (WWC, 2013), which take into account the size of the effect, statistical significance, and the status of contrary evidence. Second, we will interpret the results in light of the methodological qualities of the primary studies upon which this review is based. Finally, we will connect and compare the results with prior meta-analyses of the same or similar topics.

Treatment of Qualitative Studies

Although we will not formally synthesize qualitative studies, we will gather some qualitative information from all quantitative studies included and a selective number of well-conducted qualitative studies. More details of such qualitative information is described in the Secondary Outcomes above. This information will be included in the data for the Secondary Outcomes.