PROTOCOL: Key characteristics of effective preschool-based interventions to promote self-regulation: A systematic review and meta-analysis

Dosage

Dosage, traditionally understood as the planned amount of training administered during an intervention, plays a crucial role in understanding how interventions can be optimally delivered, resourced, replicated, and scaled up (Rowbotham et al., 2019; Wasik et al., 2013). Dosage also captures the notion of “the change to amount dispensed over time,” without necessarily implying linear causal assumptions (Rowbotham et al., 2019, p. 1).

Wasik et al. (2013) distinguish between two forms of dosage: intervention dosage and implementation dosage. Intervention dosage refers to the planned volume of training intended for the target group, as specified in the study design. In contrast, implementation dosage accounts for the actual volume of training delivered and received, influenced by factors such as adherence to intervention protocols (Musci et al., 2019). Implementation dosage has been shown to predict outcomes such as teacher adherence (Meza et al., 2020) and student engagement (Laurent et al., 2019).

Our review examines how intervention effects may vary based on both types of dosage. When we use the term “dosage,” we refer to “intervention dosage,” in accordance with the intention-to-treat principle (McCoy, 2017). Additionally, we intend to examine the influence of implementation levels on these effects.

By taking into account both forms of dosage, we strive for a nuanced understanding of the intervention's effectiveness and its applicability in real-world settings. This dual focus enables us to interpret the outcomes of the intervention from both a design and practical implementation standpoint.

Limited yet significant evidence exists regarding the relationship between dosage and intervention outcomes. For instance, some studies indicate that higher dosages may be more effective in interventions targeting executive function (Davis et al., 2007; Diamond, 2012; Tang et al., 2012; Watson et al., 2017). Research on mindfulness-based interventions has also found a positive relationship between training duration and the efficiency of the executive attention network in relation to self-regulation (Tang et al., 2007, 2009). However, it is worth noting that Tang et al.'s (2012) findings were based on undergraduate students in the US, and caution should be exercised when generalizing these findings to different demographics, such as preschool children. Additionally, Davis et al. (2007) found that overweight nine-year-olds who participated in 40-min exercise sessions five days a week for 15 weeks showed greater improvements in executive functions compared to their counterparts who exercised for only 20 minutes with the same frequency.

Group size

The impact of group size on intervention effects is a subject of ongoing debate. While smaller groups are generally favored for their potential to offer more individualized support, feedback, and opportunities for relationship-building (Solheim & Opheim, 2019), the research findings are not universally conclusive. For instance, some studies suggest that teachers may not significantly alter their teaching practices in smaller classes, thereby casting doubt on the efficacy of reducing class size as a strategy for improved learning outcomes (Hattie, 2009). Another study indicated only a small effect of class size on reading achievement and a negligible effect on mathematics achievement (Filges et al., 2018).

However, classroom dynamics are influenced by more than just the number of students. To fully grasp the implications of class size on learning outcomes, it is necessary to look more closely at its interplay with key classroom processes such as student engagement, relationships with classmates, instructional practices, and classroom management (Blatchford & Russell, 2020).

Recognizing the potential significance of class size in the context of preschool self-regulation interventions, we plan to explore this aspect in our meta-analysis. Given the nuanced and context-dependent nature of the debate surrounding classroom size, this review refrains from taking a definitive stance but emphasizes the need for additional empirical research.

While the existing literature is inconclusive, both dosage and group size could be important factors influencing the effects of preschool self-regulation interventions. To provide a more comprehensive understanding, our review will employ meta-regression analyses that consider the following intervention characteristics under Resource Allocation: ◆

Period: The length of the intervention in days or weeks

Theoretical foundations

Interventions anchored in self-regulation theory have been shown to significantly impact their outcomes. Zimmerman's three-phase model of self-regulated learning—encompassing preparation, performance, and appraisal—is a prevalent framework in primary and secondary school interventions (Panadero, 2017). Meta-analyses reveal that interventions employing social-cognitive theory or a blend of social-cognitive and metacognitive theories produce the most substantial effects, while those based on motivational theories demonstrate more modest effects (Dignath et al., 2008; Dignath & Büttner, 2008). Self-Determination Theory offers another perspective on self-regulation, conceptualizing it as goal selection in harmony with individual needs and values (Day et al., 2022). Moreover, Vygotsky's socio-cultural perspective, which underlies the Tools of the Mind curriculum, provides valuable insights into self-regulation (Barnett et al., 2008). Nonetheless, the effectiveness of specific interventions in real-world settings can vary, highlighting the necessity for continued research.

Activity variants

Physical activities, notably those requiring a blend of working memory, inhibition, and shifting, are shown to enhance executive function and self-regulation in young children (Becker et al., 2014; Diamond, 2012). An interesting avenue of research explores active play during outdoor preschool recess, revealing that it contributes positively to self-regulation, emergent literacy, and math skills (Becker et al., 2014). These activities appear to enhance academic achievement, with self-regulation playing a moderating role.

Music-based activities provide a conducive context for self-regulatory growth (Williams, 2018; Williams & Berthelsen, 2019; Zachariou & Whitebread, 2015, 2017, 2019). Combining music play with rhythmic body movements has been observed to foster self-regulation through improved beat synchronization, motor coordination, relaxation, emotional regulation, and executive function (Williams, 2018; Williams & Berthelsen, 2019). Studies conducted in the UK and Cyprus lend empirical support to these benefits (Zachariou & Whitebread, 2015, 2019).

Furthermore, open-ended activities are defined as activities without a fixed or predetermined outcome, allowing children the freedom to explore, create, and learn in a flexible environment. Examples of such activities include pretend play and construction play, which are naturally engaging for children and serve as effective platforms for developing self-regulation (Berk et al., 2006; Berk & Meyers, 2013; Braund & Timmons, 2021; Whitebread & O'Sullivan, 2012). Compared to more structured activities with predetermined goals, open-ended activities have been found to be particularly beneficial in fostering verbal self-regulation. Additionally, storybooks serve as another form of open-ended, child-directed activity, offering opportunities for pretend play that fosters exploration, expression, and the learning of self-regulatory strategies (Rowe, 1998; Welsch, 2008).

Mindfulness training has shown promise in strengthening self-regulation by enhancing the mind-body connection. The benefits can be amplified when combined with physical exercise (Diamond & Lee, 2011; Razza et al., 2015; Tang et al., 2012).

Lastly, academic activities with embedded strategy instruction can be advantageous for boosting self-regulation in a school context. This approach aligns with the social cognitive view of learning, focusing on observation, emulation, and self-reflection (Bandura, 1977, 1986; Schunk & Zimmerman, 2007). It also fits with Panadero's proposed framework for designing self-regulated learning interventions, focusing on the preparation, performance, and appraisal phases (Panadero, 2017).

Overall, activities aimed at fostering self-regulation in preschoolers come in various forms and are informed by diverse theoretical frameworks. To provide an integrated understanding of how these activities and frameworks influence self-regulation outcomes, our meta-regression analyses will investigate the following specific characteristics under Activity Type: ◆

Social Cognitive Theory-Based Activities (e.g., Zimmerman's three-phase model of self-regulation)

Years considered

We will not exclude studies by year of publication.

Language

We will include studies that were written in English.

We will exclude studies written in languages other than English.

Publication status

We will include empirical studies that report primary data obtained first-hand through the data collection (Sindin, 2018). Eligible studies may be published (e.g., journal articles, book chapters, conference proceedings) or unpublished (e.g., dissertations) literature.

We will exclude reviews, conceptual papers, introductory book chapters, or other sources that do not contain primary data.

Study designs

We will include the following interventional study designs that allow for causal inference: 1.

Randomized Controlled Trial (RCT): a.

Standard (parallel) RCTs

We consider randomized controlled trials (i.e., RCTs), in which units are randomly assigned to an intervention (treatment) group, a comparison group, or a control (business-as-usual) group, to be the optimal study design for obtaining unbiased estimates of intervention effects (Reeves et al., 2023). The difference between standard and cluster-RCTs lies in the unit of randomization. Standard RCTs use individuals as the unit of randomization, whereas cluster-RCTs use groups of individuals as the unit of randomization. Crossover RCTs also use randomization, although the initial group assignment is switched mid-study so that the same participants undergo both intervention and control conditions in two consecutive phases. The strength of crossover RCTs is their efficiency. Compared to standard RCTs with a simple parallel-group design, crossover RCTs require fewer participants because each participant acts as their own control group (J. P. T. Higgins, Eldridge, et al., 2023). However, crossover RCTs may not be suitable for self-regulation interventions as there may be carry-over effects between phases, which we will avoid by extracting data only from the first phase.

Non-randomized controlled studies of intervention (i.e., NRSIs) inherently carry a greater risk of bias (Ferriter & Huband, 2005; J. A. Sterne et al., 2023). However, we have opted to include NRSIs in our review for several reasons. First, due to the limited number of available RCTs, incorporating NRSIs can enrich our understanding of the current state of evidence concerning self-regulation interventions. Second, high-quality NRSIs can approximate the rigor of RCTs in certain contexts (Ferriter & Huband, 2005). Additionally, NRSIs often offer greater external validity, allowing for broader generalization of the findings to real-world settings. Among NRSIs, two study designs are considered particularly relevant: quasi-RCTs and non-RCTs (Cochrane Effective Practice and Organisation of Care, 2017; Reeves et al., 2017, 2023). In both of these designs, control over participant allocation is in the hands of the investigator. Quasi-RCTs employ a quasi-random method of allocation (e.g., based on participants’ birthdays), while non-RCTs use a non-random method. There is some debate in the literature about whether controlled before-after studies (i.e., CBA studies) should be distinguished from non-RCTs. CBA studies do not involve active group assignments by researchers (W.-P. Schmidt, 2017). However, some researchers, such as Polus et al. (2017), argue that this distinction is artificial and impractical, often due to poor reporting. In light of this, we will consider specific study design features when assessing the risk of bias but will not make a distinction between CBAs and non-RCTs.

We will exclude study designs that use difference-in-differences analyses and interrupted time series, as these methods are most commonly used in natural experiments where interventions can be explored but are not under the investigator's control (Craig et al., 2012; Polus et al., 2017). Our focus is on controlled experiments where the investigator designs, implements, and evaluates interventions targeting children's self-regulation in the preschool classroom. We will also exclude studies that use instrumental variables and regression discontinuity for the same reason. These methods reflect the treatment effect only for a subgroup of the population, not everyone in the sample, and are known to produce larger estimates than the intention-to-treat approach (Angrist, 2006), which we will address in this review. Furthermore, we will exclude other intervention studies that do not control for confounding factors (e.g., uncontrolled before-after studies), use mediation, latent growth, or cross-lagged analyses without reporting pre-and post-intervention outcomes for the intervention and control groups, or use only a qualitative method of data collection and analysis (Noyes et al., 2023). Finally, we will exclude observational studies, such as cross-sectional studies, or other studies that do not assess the effects of interventions on child outcomes.

Interventions

We will include universal or tier-one interventions that focus primarily on promoting self-regulation or self-regulated learning in preschool children. Interventions can be of any duration and can be delivered by either school staff (e.g., preschool or kindergarten teachers) or outside experts (e.g., researchers) as long as the interventions can be readily implemented by teachers with minimal training (McClelland & Cameron, 2012; Zhou et al., 2012). For example, extensive mindfulness or music practice requires teachers with such expertise, so we will exclude these interventions. In addition, we will look for direct interventions in the form of tasks or activities that are specifically designed to improve children's self-regulation, while teachers can be trained to effectively implement the intervention. Interventions may also target other outcomes of interest, but the focus must be on self-regulation. Moreover, we will include interventions that target executive function in relation to this criterion. This is because some interventions targeting executive function (1) train not only discrete components of executive function, but also self-regulation (e.g., integrative executive function), and (2) include relevant measures of self-regulation.

We will exclude interventions unrelated to school activities, such as self-regulation of eating or health behaviors, interventions that require expertise and extensive training (e.g., occupational therapy), and prepackaged interventions that were not originally intended to promote self-regulation, including contemplative practices (e.g., mindfulness and meditation; Flook et al., 2015), sports (e.g., martial arts; Lakes & Hoyt, 2004), music (Shen et al., 2019), literacy (Cavanaugh et al., 2017), or mathematics (DeFlorio et al., 2019). Nevertheless, we will include interventions that selectively incorporate some elements of such practices (e.g., academic tasks or mindfulness, musical, or physical activities) into activities primarily designed to promote self-regulation.

Moreover, to focus on the sources of heterogeneity according to intervention characteristics of interest (i.e., Resource Allocation, Activity Type, and Instructional Method), we will exclude preschool-based interventions that aim to indirectly influence children's social environment by creating a favorable classroom climate (e.g., the Conscious Discipline program), the teacher-child relationship (e.g., the Chicago School Readiness Project), and professional development or parent training to improve regular classroom practice or child-rearing (that goes beyond the training required to implement the intervention; e.g., the Research-based Developmentally Informed Parent program or REDI-P).

Finally, we will exclude complex interventions such as interventions that combine direct and indirect causal pathways to develop self-regulation (i.e., a combination of child, teacher, and/or parent training) or interventions that are integrated into (and thus inseparable from) school curricula (e.g., Tools of the Mind). Although Tools of the Mind focuses on developing self-regulation through structured dramatic make-believe play, the program takes a holistic approach to promoting multiple domains of child development (e.g., academic skills and socio-emotional development, including self-regulation) as a comprehensive curriculum (Baron et al., 2017, 2020; Bierman & Torres, 2015). Therefore, the effectiveness of the Tools curriculum depends on nonlinear interactions between the key components of Tools and the context under study. The complexity of the intervention makes it difficult to attribute observed effects to characteristics of the intervention (N. C. Campbell et al., 2007; Craig et al., 2008; Pigott & Shepperd, 2013). For the same reason, we will exclude other existing educational programs or curricula such as Montessori education (Ervin et al., 2010; Lillard, 2012; Lillard et al., 2017), the Promoting Alternative Thinking Strategies (PATHS) curriculum (Morris et al., 2014), the Head Start Research-based, Developmentally Informed (REDI) intervention (Bierman et al., 2008, 2014), the Chicago School Readiness Project (CSRP; Jones et al., 2013; Raver et al., 2011), and Conscious Discipline (K. L. Anderson et al., 2020). Similarly, we will exclude interventions that target self-regulation as part of a broader set of abilities (e.g., school readiness, socio-emotional skills, critical thinking, understanding and expressing emotions, and Theory of Mind), although this decision often requires a review of the full text to confirm what the authors mean by these terms.

Setting

We will include interventions that take place in preschools, defined as formal out-of-home education and care that children attend before entering primary school (Dietrichson et al., 2020). Preschools may also be referred to as pre-primary schools, play schools, kindergartens, nursery schools, daycare centers, and pre-kindergartens. Note that some preschool programs may be housed on primary school campuses. Although we will not exclude preschool-based interventions that are combined with another intervention outside of the school setting under this criterion (e.g., parent training at home), we will exclude complex interventions under the criterion for interventions.

We will exclude interventions that take place entirely outside the preschool setting (e.g., foster care, nannies, or parent training at home), except for summer programs to prepare children for kindergarten whose target population and intervention characteristics are sufficiently similar to the preschool-based interventions we examine. We will also exclude computer-mediated interventions (e.g., training that incorporates information and communication technology such as computers or tablet apps).

Primary outcomes

We will include measures of the complex self-regulatory processes: self-regulation and self-regulated learning. Measures of self-regulation include performance-based measures such as the Preschool Self-regulation Assessment (PSRA; Smith-Donald et al., 2007) and the Preschool Situational Self-Regulation Toolkit (PRSIST) assessment (Howard et al., 2019) or questionnaires such as the Child Behavior Rating Scale (CBRS; Bronson et al., 1990) and the Child Self-Regulation and Behaviour Questionnaire (Howard & Melhuish, 2017). Self-regulated learning (i.e., self-regulation of learning) can be assessed through various phases of problem-solving tasks (e.g., planning, monitoring, and evaluating) using the C. Ind. Le Coding Framework (i.e., the observational coding framework for verbal and non-verbal indicators of metacognitive and self-regulatory processes in children aged three to five; Whitebread, Coltman, Pasternak, et al., 2009), the Children's Independent Learning Development checklist (CHILD 3-5; Whitebread, Coltman, Pasternak, et al., 2009), and the Train Track Task, which captures a metacognitive aspect of self-regulated learning (Bryce et al., 2012; Bryce & Whitebread, 2015).

In addition, we will include performance-based measures of basic self-regulatory processes: integrative executive function and integrative effortful control. More specifically, we are interested in measures that assess the active integration of components of executive function (i.e., working memory, inhibition, and shifting; Hofmann et al., 2012; McClelland, Cameron, Connor, et al., 2007; McClelland, Cameron, Wanless, et al., 2007; Rueda et al., 2005) or effortful control (i.e., delaying gratification, gross motor control, fine motor control, suppress-initiate response to signals, and effortful attention; Kochanska & Knaack, 2003; Murray & Kochanska, 2002; Zhou et al., 2012) within a task. For executive function measures, we will include Heads-Toes- Knees-Shoulders (McClelland & Cameron, 2012), the Hearts and Flowers task (Wright & Diamond, 2014), the Dots test (or task; Davidson et al., 2006; Diamond et al., 2007), and the Minnesota Executive Function Scale (Carlson & Zelazo, 2014), which require children to pay attention, use working memory to remember the instruction, and use inhibitory control to respond to the task, despite potential ecological validity issues (Hammer, 2018). Because our previous knowledge of such measures is limited, particularly for integrative effortful control skills, we will include other measures that meet this criterion.

Accordingly, we will exclude studies that do not measure basic and complex self-regulatory processes at the child level. In addition, we will exclude measures of discrete components of executive function or effortful control (McClelland & Cameron, 2012) or measures of discrete executive functions that are grouped together as global executive function. These include the Behavior Rating Inventory of Executive Function—Preschool Version (Sherman & Brooks, 2010), the Early Years Toolbox (Howard & Melhuish, 2017), and the NIH Toolbox Cognition Battery (Zelazo et al., 2013).

Secondary outcomes

Secondary outcomes include all quantitative measures of academic skills (i.e., emergent literacy and math skills).

Data collection process

AK has developed a comprehensive set of coding instructions for both the study level and the effect size level to guide the data extraction process (see Supporting Information: Appendix 3 for coding instructions). These instructions were tested and subsequently refined in consultation with IS. This approach aims to ensure consistency and transparency among reviewers during data extraction, thereby minimizing the need for frequent reference to original data sources during both data synthesis and risk of bias assessment (Li et al., 2023).

For data collection, two reviewers will be assigned to extract data from each eligible study. Due to the review's extended timeline and varying reviewer availability, we cannot specify the exact number of reviewers for this phase. Only those reviewers assigned to this task will undergo training, which includes watching an instructional video, participating in calibration exercises, and engaging in discussions to resolve any ambiguities related to data collection procedures.

AK will take the lead in data extraction, using a standardized grid to collect data from all included studies. The role of the other reviewers will involve verifying the accuracy of AK's study-level coding and independently performing effect size-level coding for their allocated sections of the included studies.

Any disagreements that arise during this process will be resolved through discussion among the reviewers. If additional clarification is needed, we will not hesitate to contact the authors of the studies in question.

We will extract the following study characteristics:

Study-level coding

1.

Bibliographic information: date of extraction; report ID; study ID; publication type; author; publication year; study title

Effect size-level coding

1.

Outcome: a.

Self-regulation: construct; self-regulation system; domain specificity; the name of the measurement; measurement type

Primary outcomes

We will differentiate outcomes in self-regulation based on their operational definitions and types of measurement.

First, we will consider four distinct approaches to conceptualizing child self-regulation: these include self-regulation, self-regulated learning, executive function, and effortful control. While executive function has traditionally been the focus of cognitive neuroscience and clinical psychology, primarily in contexts devoid of emotional influence, effortful control has been examined within the realm of temperament research, particularly in emotionally charged settings (Zhou et al., 2012). Despite these divergent research traditions, Zhou and colleagues highlight several areas where the definitions and operational aspects of executive function and effortful control overlap. They advocate for a unified model that integrates these two theoretical frameworks.

Second, we will categorize self-regulation into two primary systems: the cognitive (“cool”) system, which focuses on cognition and behavior, and the affective (“hot”) system, which centers on motivation and emotion (Dent, 2013; Zhou et al., 2012). Additionally, we will consider an integrated approach that combines both cool and hot dimensions of self-regulation. Prior research has established a connection between the development of executive function and cool self-regulation with academic achievement, while effortful control and hot self-regulation have been associated with socio-emotional development (McClelland & Cameron, 2012; Willoughby et al., 2011; Zhou et al., 2012).

Third, we will differentiate between domain-general self-regulation, which refers to foundational abilities applicable across various life contexts, and domain-specific self-regulation, which focuses on abilities tailored to particular settings or subjects such as academics or social interactions (Gunzenhauser & Saalbach, 2020). This distinction is vital for assessing the scope and applicability of self-regulatory interventions, as it allows us to understand whether the abilities developed are broadly transferable or more targeted within specific domains.

Fourth, we will distinguish between online and offline measures of self-regulation based on the timing of the data collection (Araka et al., 2020; L. Jacob et al., 2019; McClelland & Cameron, 2012; Rovers et al., 2019; Schmitt et al., 2015). Online measures collect data during the execution of the actual learning task, whereas offline measures collect data before and after performance. Specifically, online measures tend to assess ongoing specific self-regulatory behaviors or strategies as events or states, whereas offline measures are more inclined to assess children's self-regulation as aptitude or traits or global use of strategies through reflection (Inzlicht et al., 2021, p. 20; Rovers et al., 2019; Winne, 2010).

Fifth, researchers have increasingly noted the ecological validity of direct measures of self-regulation across contexts compared to indirect measures such as teacher reports or classroom observations (McClelland et al., 2012; McClelland & Cameron, 2015; Schmitt et al., 2015). Overall, we expect that measures that rely heavily on preschoolers’ verbal skills or reflection including thinking aloud, self-reports, or structured interviews, will be used less frequently (L. Jacob et al., 2019; Whitebread, Coltman, Pasternak, et al., 2009). We plan to examine heterogeneity in the summary effect of the intervention using the four measurement types explained above. In addition, we will use multiple measures of self-regulation within a single study.

Secondary outcomes

We will include all quantitative measures of children's academic skills (e.g., emergent literacy, math skills) as secondary outcomes.

Timing of the assessment

We will include data collected during the short-term (up to five months post-intervention), medium-term (six months to 11 months post-intervention), and long-term (12 months or more post-intervention) follow-up periods as secondary outcomes.

Parallel and crossover RCTs

For parallel RCTs that use an appropriate randomization method, we will use summary statistics so that intervention effects can be estimated consistently across studies (J. P. T. Higgins, Li, et al., 2023). Similarly, for crossover RCTs, we will use summary statistics from the first trial period (J. P. T. Higgins, Eldridge, et al., 2023). More specifically, we will first calculate Hedges’ g from the change-from-baseline scores (i.e., the difference between pre-and post-intervention scores) for parallel RCTs and crossover RCTs using formula 1 (i.e., a combination of equations 5, 27, 28, and 29 in Wilson, 2017; see Figure 4), where g is Hedges’ g, M is the mean for each group, SD is the standard deviation, n is the size of each group, ω is a small sample size bias correction factor (What Works Clearinghouse, 2022), df is the degrees of freedom (What Works Clearinghouse, 2022), N is the total sample size, and the subscripts 1, 2, 3, and 4 represent treatment pre-intervention, treatment post-intervention (or follow-up evaluation), control pre-intervention, and control post-intervention (or follow-up evaluation), respectively.

OPEN IN VIEWER

Figure 4

Formulas for calculating Hedges' g in RCTs.

We will calculate SE _[g] (i.e., the standard error of g) using formula 4 (What Works Clearinghouse, 2022; see Figure 5), where n is the size of each group and the subscripts t and c represent the treatment and control groups, respectively.

OPEN IN VIEWER

Figure 5

Formula for calculating the standard error of Hedges' g.

SE _[g] is squared to produce V _g (i.e., the variance of g) using formula 5 (see Figure 6).

OPEN IN VIEWER

Figure 6

Formula for calculating the variance of Hedges' g.

Quasi-RCTs and non-RCTs

For quasi-RCTs and non-RCTs, we will prioritize extracting adjusted effect estimates to reduce bias due to confounding and missing data (e.g., the Full Information Maximum Likelihood Estimation; Enders, 2001). Therefore, we will attempt to convert available cluster-adjusted effect sizes (e.g., Cohen's d) and variance (e.g., V _d , which can also be calculated from the confidence interval) into Hedges’ g and its variance (V _g ) or use unstandardized regression coefficients representing an intervention effect to calculate g using formula 6 (What Works Clearinghouse, 2022; see Figure 7), where g is Hedges’ g, ω is a small sample size bias correction factor for individual-assignment studies, b is an unstandardized regression coefficient, SD is the standard deviation, and n is the size of each group.

OPEN IN VIEWER

Figure 7

Formula for converting effect sizes to Hedges' g in quasi-RCTs and non-RCTs.

If quasi-RCTs and non-RCTs report standardized regression coefficients (β), we will transform β into b. To do this, we will first calculate the standard deviation of the independent variable (S _x ) using formula 7 (Wilson, 2017; see Figure 8).

OPEN IN VIEWER

Figure 8

Formula for calculating the standard deviation of the independent variable in quasi-RCTs and non-RCTs.

S _x is used to calculate an unstandardized regression coefficient (b) using formula 8 (Wilson, 2017; see Figure 9).

OPEN IN VIEWER

Figure 9

Formula for transforming standardized regression coefficients (B) to unstandardized (b) in quasi-RCTs and non-RCTs.

Subsequently, b is used to calculate Hedges’ g with formula 6.

We will produce SE _[g] for quasi-RCTs and non-RCTs in two ways. First, if we can assume that a study's statistical model produced intervention effect estimates that were properly adjusted for covariates, we will use the reported R ² in calculating SE _[g] using formula 9 (What Works Clearinghouse, 2022; see Figure 10).

OPEN IN VIEWER

Figure 10

Formula for calculating the standard error of Hedges' g using reported R² in quasi-RCTs and non-RCTs.

Second, when studies do not report R ² values, we will rescale the standard error of a correct model using formula 10 (What Works Clearinghouse, 2022; see Figure 11), where we use a standard error (which can also be calculated from a confidence interval) of the unstandardized regression coefficient for an intervention effect (i.e., SE _[b] ) and a pooled sample standard deviation (i.e., SDp; see formula 11) to produce SE _[g].

OPEN IN VIEWER

Figure 11

Formulas for rescaling standard error using standard error of unstandardized coefficient and pooled sample standard deviation in studies without reported R² .

When studies report the standard error for a standardized regression coefficient (SE _[β] ), we will calculate the standard error for the unstandardized regression coefficient (SE _[b] ) using formula 12 (see Figure 12).

OPEN IN VIEWER

Figure 12

Formula for calculating standard error of unstandardized regression coefficient from standardized coefficient.

Then, SE _[b] is used to produce SE _[g] with formula 10.

If adjusted effect estimates are not available (and only summary statistics are available) in the study reports, we will calculate Hedges’ g and its variance (V _g ) using formulas 1, 2, 3, 4, and 5, taking into account the potential bias of the calculated effect estimates (see the section “Assessment of risk of bias”). Accordingly, we will record the details of the covariates, with particular attention to those that may act as confounders, and the methods used to account for missing data.

Cluster RCTs (and other studies with clustering)

Cluster-RCTs should take into account the clustering of individuals in addition to the biases introduced by confounding and missing data. If study authors properly account for these biases, we will primarily use model-based effect estimates extracted directly from study reports (typically from a linear regression of cluster-specific mean outcomes on randomized groups, weighted by cluster size; I. R. White & Thomas, 2016). In other words, we will attempt to convert available cluster-adjusted effect sizes (e.g., Cohen's d) and variance (e.g., V _d , which can also be calculated from the confidence interval) into Hedges’ g and its variance (V _g ) or use an unstandardized regression coefficient representing an intervention effect with formulas 2, 13, 14, and 15 (What Works Clearinghouse, 2022; see Figure 13), where γ is a small number of clusters adjustment term, n is the average cluster size (i.e., the total number of individuals divided by the total number of clusters), and ρICC is an intraclass correlation coefficient.

OPEN IN VIEWER

Figure 13

Formulas for adjusting cluster-RCT effect sizes and variance to Hedges' g and its variance, including intraclass correlation adjustment.

As with quasi-RCTs and non-RCTs, we will transform standardized regression coefficients (β) into unstandardized regression coefficients (b) using formulas 7 and 8.

We will prioritize obtaining standard errors accounting for covariates and clustering in three ways. First, when study authors report a cluster-corrected standard error (which can also be calculated from a confidence interval) for an unstandardized regression coefficient that represents an intervention effect (SE _cc[b] ), we will produce SE _[g] using formula 16 (What Works Clearinghouse, 2022; see Figure 14), where SD _p is defined in formula 11, γ is defined in formula 15, and df is defined in formula 14.

OPEN IN VIEWER

Figure 14

Formula for calculating standard error of Hedges' g with cluster-correction and covariate adjustment.

Second, if cluster-RCTs account for covariates but not clustering, we will incorporate a design effect term (η; see formula 18 for its definition) to adjust standard errors not corrected for clustering (SE _uc[b] ) using formula 17 (What Works Clearinghouse, 2022; see Figure 15).

OPEN IN VIEWER

Figure 15

Formulas for adjusting standard errors with design effect in cluster-RCTs not correcting for clustering.

If cluster-RCTs report the standard error for a standardized regression coefficient (i.e., SE _cc[β] or SE _uc[β] ), we will calculate the standard error for the unstandardized regression coefficient (i.e., SE _cc[b] or SE _uc[b] ) using formula 12. Then, SE _cc[b] or SE _uc[b] is used to produce SE _[g] with formula 16 or 17, respectively.

Third, if studies report R² from a single-level model (e.g., ANCOVA or ordinary least-squares regression), rather than reporting the standard error of the regression coefficient adjusted for covariates or clustering, we will use R ² and η to produce SE _[g] using formula 19 (What Works Clearinghouse, 2022; see Figure 16).

OPEN IN VIEWER

Figure 16

Formula for calculating standard error of Hedges' g using R² and design effect in single-level model analyses.

We will apply the above calculation to quasi-RCTs and non-RCTs with clustering of individuals. If adjusted effect estimates are not available in the study reports, we will make full use of the available information to calculate Hedges’ g and its variance (V _g ) adjusted for either confounding, clustering, or missing data. Studies with clustering may not report covariate-adjusted regression coefficients, but we can calculate Hedges’ g (e.g., a combination of formulas 1, 2, 14, and 15) and V _g (e.g., a combination of formulas 2, 4, 14, and 18) adjusted for clustering from the data obtained. As with quasi-RCTs and non-RCTs, we will record the details of the covariates, the methods used to account for missing data, and the type and degree of clustering (J. P. T. Higgins, Thomas, et al., 2023; What Works Clearinghouse, 2022), and assess the risk of bias accordingly.

Calculators

For the calculations, we will use either an Excel spreadsheet with the above formulas (using full, unrounded values for numeric calculations) or an onLine software tool called Practical Meta-Analysis Effect Size Calculator (Wilson, 2023). The choice will depend on the type of data extracted from the included studies. In synthesizing effect sizes, we will align the effect direction across effect sizes such that the positive effect direction refers to the extent to which the intervention develops self-regulation. Studies reporting medians and interquartile ranges may have done so for skewed data. In such cases, it is possible to conduct a meta-analysis on a log-transformed scale, although log-transformed and non-transformed data cannot be mixed in a meta-analysis (Deeks et al., 2023).

Since we expect few studies to report medians and interquartile ranges, we will attempt to use means and standard deviations if we assume that the true distribution of outcomes in these studies is symmetric. If the above data are missing, we will either attempt to convert the data reported in the studies (e.g., statistics) to Hedges’ g, or contact the authors to obtain the necessary data. Although we cannot summarize the missing results of eligible studies in a meta-analysis, we will present these studies with available information (e.g., group sample sizes) along with the primary results of the meta-analysis. Because we are interested in the intention-to-treat effect (i.e., the effect of assignment), for studies that conduct intention-to-treat analyses, we will use the sample size at the time of enrollment in the study for estimation. However, if some participants are excluded from the analysis (i.e., available case analysis), we will use data only from participants whose outcomes are known and consider the potential impact of missing data when assessing the risk of bias.

Finally, while it is ideal to calculate summary statistics for each study in the same way (Deeks et al., 2023), this may be unrealistic due to differences in how statistics are reported in studies and the different priorities we set when extracting statistics from different study designs (i.e., summary statistics vs. adjusted effect estimates). As described above, when calculating effect estimates from the summary data, we will use the change-from-baseline scores to remove variability between-individuals in the baseline. Because the standard deviations used to standardize post-intervention scores (i.e., between-individual variability) reflect different variability than the standard deviations used to standardize change-from-baseline scores (i.e., both within-individual and between-individual variability), we will not pool SMDs calculated from post-intervention scores with those from change-from- baseline scores (Deeks et al., 2023). Nevertheless, we will combine effect estimates based on change-from-baseline scores with effect estimates that have been statistically adjusted for baseline scores (e.g., by analysis of covariance or ANCOVA) to make full use of the available data. In this regard, we will conduct sensitivity analyses to assess the impact of using different calculations to generate effect estimates (J. P. T. Higgins, Li, et al., 2023).

Analytical approach

◆

Small-study effects

We will compare studies with relatively small samples to other studies to examine whether the average effect in random-effects analysis is inflated by studies with small samples. The sample size threshold will be set depending on the studies included. ◆

Methods for calculating the SMD

Consistency in the methods used to calculate SMD is highly desirable. However, to combine the best available evidence on the review questions across a heterogeneous group of studies, we chose to combine Hedges’ g from both change-from-baseline scores (based on summary statistics) and effect estimates adjusted for confounding, clustering, and missing data. Thus, we will run three separate univariate meta-regressions to compare the results with and without adjustments for confounding, clustering, and missing data. Confounding and missing data may bias the summary estimates and reduce their precision. On the contrary, because the studies that ignore clustering tend to receive more weight than they should, the results that ignore clustering may have artificially narrower confidence intervals around the effect estimates (J. P. T. Higgins, Thomas, et al., 2023).

Study backgrounds

◆

Study design

We will conduct a univariate meta-regression to test the validity of the decision to combine the results of RCTs and NRSIs in a meta-analysis. If we find systematic differences in effect estimates between RCTs and NRSIs, we will conduct a multivariate meta-regression of the summary estimate with Study design (RCTs vs. NRSIs), Risk of bias (lower risk of bias vs. higher risk of bias), and interaction term (i.e., Study design × Risk of bias) to examine whether these differences can be attributed to differences in risk of bias between RCTs and NRSIs or to the lack of directness of NRSIs in answering the review questions. ◆

Risk of bias

We will use univariate meta-regression to compare the results of meta-analyses that include or exclude studies at high risk of bias. If we find that the decision to include the studies at high overall risk of bias results in an invalid summary effect estimate and a substantial increase in heterogeneity, we will reduce confidence in the conclusion of the meta-analysis. Based on the sensitivity analysis of the effects of study design and risk of bias, we will discuss the key features of the study design that may lead to valid estimates of intervention effects in future research.

Outlying studies

To evaluate the influence of potential outliers on our meta-analysis, we will conduct dedicated sensitivity analyses that include and exclude these studies. Outliers will be systematically identified using statistical diagnostics such as Cook's distance and leverage values, as well as through visual inspection of forest and funnel plots. Upon identification, we will assess their impact on key metrics such as the overall effect size and between-study heterogeneity. If the inclusion of outliers substantially distorts these metrics, they may be excluded from the main analysis. The outcomes of these sensitivity analyses will be meticulously documented and reported to offer a nuanced understanding of the robustness of our meta-analytic findings and provide insights into the stability of the results when subjected to the influence of outlying data points.

In addition, we plan to conduct univariate meta-regressions to examine the consistency of results across subgroups of the following study characteristics:

Operationalization

◆

Self-regulation construct (i.e., self-regulation vs. self-regulated learning vs. the integration of executive functions or effortful control)

Participant characteristics

◆

Country

In the meta-regressions, we will not adjust the significance level because our goal is to informally assess how different methods, assumptions, definitions, and subgroups may affect the conclusions of the meta-analyses based on the magnitude and direction of the effect estimates without relying solely on statistical significance (Page et al., 2021; Thabane et al., 2013).