Montessori education's impact on academic and nonacademic outcomes: A systematic review

Montessori interventions

Although the included studies designated the intervention condition as “Montessori,” the studies' methods sections provide varying levels of description of the intervention and although most of those descriptions indicated that the intervention had the philosophical elements of Montessori education, some revealed that they deviated in certain ways from the structural ones. Because of this variation, in response to the first set of reviews we categorized (post-hoc) the Montessori conditions into five categories. All of the Montessori interventions appeared sensitive to the philosophical elements of Montessori, for example touching on free choice and hands-on materials in the article Introduction if not also in Methods. Where they varied the most was in structural elements.

Although we performed this categorization, we do not advise considering how effect sizes might vary with the implementation levels because effect sizes stem from many sources including the different effectiveness of the Montessori intervention relative to its control condition; the control or traditional conditions also varied, as they do in the real world. A second reason not to consider these levels for analysis is that the variables that led to the categorizations are our best estimates based on what was reported; they lack precision. In sum, we provide the levels and their descriptions here only to give readers a sense of the range of implementations that were considered in the meta-analysis.

At the highest level was full implementation, meaning that the school was recognized by a respected association like the Association Montessori Internationale (AMI) or the Swiss Montessori Association or at least had teachers who were fully trained by AMI or the American Montessori Society before the intervention took place. AMI recognition entails the structural elements of having AMI-trained teachers (trained to implement the philosophy to a high degree), a specific 3-year-age range in each classroom (e.g., children ages three to six or six to nine), a 2.5–3 h uninterrupted work period during which children are free to choose their own work, large class sizes and few adults, and a full set of Montessori materials. There are also no grades; the materials are self-correcting. The Montessori condition in 10 of the 32 studies appeared to meet these criteria (Denervaud, 2019, 2020; Elben, 2015; Lillard, 2006; Lillard, 2012; Lillard, 2017; Mix, 2017; Rathunde, 2005a; Rathunde, 2005b; Yussen, 1980).

The next highest level, observed in six studies, was medium implementation; for these, the article mentions some accreditation or teacher training, but the accrediting organization was unspecified (“a local Montessori association,” Prendergast, 1969), and teacher training was done by a respected organization like AMS but not all teachers had completed the training at the time of data collection (Mallett, 2015). In one case the description of Montessori suggested good implementation, but photos included in the article incorrectly called some commercial toys “Montessori materials” (Faryadi, 2017), which put the study in a lower category. In another case, implementation was discussed and a rubric was developed to measure it, and the measure indicated a nontrivial level of deviance from the highest level of implementation among the schools studied (Culclasure, 2018). In another case, the UK Montessori Schools Association had accredited the Montessori schools, but they deviated by including pretend play; this may not be a major deviation but did suggest not fully implementing Montessori (Kirkham, 2017). One study considered to have a Montessori condition at this medium level had AMS-trained teachers or teachers in training but used only two ages of children per class instead of three (Manner, 1999).

At the next level, observed in seven studies, Montessori was claimed to be implemented as a full classroom program, but there was at least one very serious deviation from the full implementation of the structure, and other elements were unspecified or suggested other problems. For five of these, classrooms had only one age level: 4-year-olds (Ansari, 2014; Galindo, 2014; Jones, 1979; Miller, 1983; Miller, 1984); three of these also had teachers with only a 6-week training course. For one study, it was stated that some classrooms had mixed grades and it specified two grades, suggesting some other classrooms had only single ages (Besançon, 2013). Another study at this level called the intervention modified Montessori and stated that some children had group instruction during part of the session (Coyle, 1968; description taken from Concannon, 1966). These all suggest some serious deviations from high-fidelity Montessori.

There were five studies in which the level of Montessori implementation could not be determined: Fleege (1967), Kayili (2016a), Kayili (2016b), Tobin (2015), Aydoğan (2016). In these studies, Montessori philosophy was described appropriately, but there was not enough information about the classrooms themselves to determine whether the structural elements were in place; one was left with very little sense of how Montessori was actually implemented. In the fifth study in this category, not only was the Montessori implementation unspecified, but the intervention appeared to occur over a 7-week period (Aydoğan, 2016); it is unclear if children had had Montessori programming before the 7-week-long study.

Finally, there were four studies in which the Montessori condition appeared to be an add-on curriculum implemented for a limited period of time. Alburaidi (2019) set up a school Science Hall that had six centers offering hands-on materials, choice, meaningful activities, order, and other Montessori characteristics; intervention students spent their 45-min science classes covering specific topics in this hall with a teacher taught to guide in a Montessori manner. Doğru 2015 taught children with ADHD to use Montessori sensorial materials (e.g., tactile boards, binomial cubes) over 8 weeks, for 45 min each week. Hoseinpoor (2014) offered 12 sessions with a variety of sensorial activities and role-play activities; it appeared that these built on each other and that each session included the activities from prior sessions. From their introduction, it seems the intervention children were free to choose among these activities. The final study in this group (Juanga, 2015) implemented Montessori for a single day using assigned workstations; the environment was prepared, the activities were hands-on, and a Montessori consultant guided the teacher.

Although there is tremendous variety in these implementations, they reflect the variety of implementations of Montessori in the real world; the term is not trademarked.

Comparison condition: Traditional education

Detailed descriptions of the comparison condition were rarely provided, as if the terms conventional or traditional made clear what they were. At the elementary school level (studies with children ages 6–12), some studies specified that the control condition had teacher-directed, whole-class learning involving textbooks, teachers checking and grading work, single-aged classrooms, and class periods of limited duration (in some cases 45 min). Most studies outside of the United States referred to a national program (such as traditional Swiss or French or Thai pedagogy, or the UK National Curriculum). At the preschool level, some non-western studies referred to (for example) traditional Malaysian or Philippine, or Iranian conventional education, without reference to a set curriculum; further details suggested these were also teacher-centered, lecture-style approaches. By contrast, for some (especially older) studies conducted in the United States, traditional education at the preschool level was described as comprised of free play and pretense (Coyle, 1968; Miller, 1983; Miller, 1984; Prendergast, 1969). One study specified that its conventional condition implemented HighScope, which involves centers offering various art, play, and learning activities; this method is modeled after the Perry Preschool Project (Ansari, 2014). Others provided little to no information about their conventional preschool condition, referring simply to the control program as pre-K, traditional, business-as-usual, or non-Montessori. In recent years preschool programs in the United States are more likely to be teacher-centered and involve little play (Bassok, 2016).

In sum, traditional and Montessori conditions reflected a range, across studies, that is reflective of the range of implementations of each in the real world. Although we categorized the Montessori implementation into levels post-hoc, we advise against considering the relation between effect sizes and these estimated levels of implementation, first because our categorization is imprecise, and second because effect sizes reflect the relative difference across Montessori and control conditions in each study, and control conditions varied as well in ways that are impossible to rank precisely.

Primary outcomes

The variety of academic outcomes was reduced to five categories of outcomes:

General academic ability. This outcome included measures that did not clearly fit one of the other categories (like math or reading), such as the cognitive subscale of the learning Accomplishment Profile-Diagnostic (Lap-D) (see Ansari, 2014); this subscale included counting as well as matching and did not report scores separately. It also included three subtests of the Bracken scale (see Galindo, 2014) including identifying colors and comparing sizes. Tests of problem solving that were not explicitly mathematical were also categorized as General Academic Ability. Finally, different academic abilities concatenated into a general academic score were included in this category.

Individually the academic outcomes had too few studies to do a moderator analysis with cluster–robust variance estimation; therefore, we also created an aggregated academic outcomes variable that comprised studies that contributed at least one effect size to one or more of the academic outcomes listed above.

Secondary outcomes

Although the nonacademic outcomes are listed as secondary outcomes here, we do not intend to assign a hierarchal structure to academic and nonacademic outcomes. After examining the variety of nonacademic outcomes, we found that there were four major nonacademic outcomes:

Creativity. This included measures like Alternate Uses, where children must come up with all possible uses for a common object like a paper clip, the Torrance Test of Creativity (which includes Alternate Uses as one of several tasks), and tests where children need to create a drawing or story and a panel of judges rates their creativity.

Individually, the nonacademic outcomes had too few studies to do a moderator analysis with cluster–robust variance estimation; therefore, we created an aggregated nonacademic outcomes variable from studies that contributed at least one effect size to one or more of the nonacademic outcomes listed above.

Databases

A comprehensive database search included the following online subscription databases:

EBSCO Academic Search Complete

Gray literature

Professional Montessori association websites, gray literature, digital repositories, and social science research websites were identified through expert referrals and Internet searches, and conference proceedings, reports, working papers, white papers, and preprints were reviewed to find relevant research. Government documents were identified primarily through the Education Resources Information Center (ERIC) and the US Department of Education's Institute of Education Sciences (IES) and various subsidiaries, projects, and digital libraries. ERIC was searched both using the EBSCOhost platform and government-hosted websites to ensure an exhaustive search. Websites were searched for relevant studies, reports, and citations. Sources for gray literature included:

American Educational Research Association (http://www.aera.net/)

Internet search

Google, Google Scholar, and Bing were used to search the open web in an attempt to fill in any gaps left after searching the specialized sources. Results were analyzed until a saturation point was reached, (i.e., until further searching led to no new articles for inclusion).

See Table 2 and Supporting Information: Appendix 1 for further information on Internet searches.

Contacting other researchers

Authors of prior studies and other experts in the Montessori method were contacted to obtain unpublished research or to get further clarification on published studies. A record of attempts to contact authors can be found in the notes section of the inclusion and exclusion data set provided in the supplemental information to this review (Randolph, 2021).

Prior reviews and reference lists

The prior reviews that were searched are mentioned in the Why it is important to do this review section and there is additional information in the supplemental information provided in Randolph (2021).

During both the search and selection processes, studies determined to be relevant enough for closer inspection of reference lists were searched for citations to additional studies. For example, a study examining Montessori and non-Montessori outcomes that had insufficient data and was therefore not included would be examined for references to other studies.

Studies with academic outcomes

Studies with academic outcomes tended to use the pretest-posttest with control group design (Shadish 2002). (Studies that used pretest-posttest without control group designs were excluded).

Studies with nonacademic outcomes

Similar to studies with academic outcomes, nonacademic studies most frequently used the pretest-posttest with control group design.

The list below provides a link to each of the 32 included studies. The supplemental information in Randolph 2021 contains a data set where we extracted the information specified in the coding book; specific details on each study can be found there. 1.

Alburaidi 2019

Main effects for academic outcomes

Table 4 and Figure 4 (a histogram of raw effect sizes for academic outcomes) summarize the main effects of Montessori education versus traditional education for academic outcomes. For readers unfamiliar with the interpretation of meta-analytic main-effects tables, we describe the interpretation of each column in Table 4 here before discussing the specific results in the following paragraphs. The first column in Table 4 indicates the outcome. The second column indicates the standardized mean difference effect size, Hedges' g, which is a sample-size corrected version of Cohen's d. Positive effect sizes favor Montessori education over traditional education, effect sizes of zero indicate the equality of Montessori and traditional education, and negative effect sizes favor traditional education over Montessori education. Since all of the effect sizes in Table 4 are positive, one can interpret the effect size as the number of standard deviations that Montessori students on average performed better than traditional students performed. See (Cohen, 1988; Kraft, 2020) for additional resources for interpreting the magnitude of an effect size. The third column indicates the 95% confidence intervals for the effect size, which can be interpreted as the plausible range of the population effect size, given chance (Higgins, 2021). The fourth column, p, is the two-tailed probability of the effect size's being equal to 0.00 given chance; a Hedges' g effect size of 0.00 (the null value) would indicate that Montessori education and traditional education are equal. Values of p below 0.05 are typically regarded as statistically significant. The fourth column indicates the df (degrees of freedom) when a cluster-robust model was used. Tanner-Smith (2014) cautioned that cluster–robust results should not be interpreted as being reliable when the df is less than 4.00. The fifth column, I ², is a measure of study heterogeneity–the difference in effect sizes among studies–expressed as a percentage. Higgins (2021, p. 259) gives the following rules of thumb for interpreting I ²:

0%–40%: might not be important;

OPEN IN VIEWER

Figure 4

Kernel density plot of unweighted values of Hedges’ g for individual academic outcomes.

Finally, the fifth and sixth columns of Table 4 indicate the number of studies and the number of effect sizes, respectively, that contributed to each outcome.

As Table 4 shows, with all academic outcomes combined across 24 studies and 113 effect sizes, Montessori students on average performed 0.24 standard deviations higher than students in traditional education programs, 95% CI [0.13, 0.36], with high study heterogeneity, I ² = 76.54%.

See Figure 5 for a graphical display of study heterogeneity (i.e., a GOSH plot) and effect sizes of academic outcomes from a bootstrap analysis of 1,000,000 samples of academic effect sizes. That GOSH plot shows the results of plotting the effect size and heterogeneity of random subsets of studies. The measure of heterogeneity (I ²) is plotted on the vertical axis and the measure of effect size is plotted on the horizontal axis. So, each data point on the GOSH plot represents the heterogeneity and summary effect size of a random sample of studies. The darker the area, the more likely it is that that is the range in which the population effect size resides. (The population effect size is the unknowable effect of Montessori education that generalizes to all Montessori programs, not just the ones investigated in the studies included here). In Figure 5, the darkest area varies horizontally between approximately 0.10 and 0.32 in effect size, so, this is the plausible range of the effect of Montessori academic outcomes. This dark area is positioned vertically at around an I ² value of 90%, indicating high study heterogeneity. On the left-hand side of Figure 5, there is a small, light tail extending down to 0% heterogeneity and with an effect size of 0.07, meaning that there is a small statistical probability that the population summary effect size is actually 0.07 with very low heterogeneity. See Olkin (2012) for more information on the interpretation of the GOSH plot.

OPEN IN VIEWER

Figure 5

Graphical display of study heterogeneity for all academic outcomes combined. In the GOSH plot above, the X-axis (Overall Estimate) shows the Hedges’ g effect size. Positive effect sizes favor Montessori education over traditional education. The Y-axis represents a measure of study heterogeneity, I ², where higher values represent more study heterogeneity. The black data points are simulated values of the population effect size and heterogeneity.

We believe there is moderate quality of evidence for this finding as discussed in the Summary of Findings table. (Note that all of our ratings of quality of evidence are based on the GRADE system for rating quality of evidence; see Higgins, 2021).

In terms of the individual academic outcomes presented in Table 4, all academic outcomes were in favor of Montessori education. In summary, there was moderate quality of evidence that Montessori education outperformed traditional education in terms of general academic ability (Figure 6) (Hedges' g = 0.26, 95% CI [0.06, 0.46]), high quality of evidence for language/literacy (Figure 7) (Hedges' g = 0.17, 95% CI [0.03, 0.31]), high quality of evidence for mathematics (Figure 8) (Hedges' g = 0.22, 95% CI [0.06, 0.39]), low quality of evidence for science (Figure 9) (Hedges' g = 0.15, 95% CI [−0.61, 0.46]), and moderate quality of evidence for social studies (Figure 10) (Hedges' g = 0.05, 95% CI [−0.01, 0.12]). Note that the three social studies outcomes only come from Culclasure (2018). See the Summary of findings Table 1 for more information on the certainty of evidence. Note that judgments of the quality of evidence throughout this review are based on the GRADE system; see the Quality of the evidence section for details.

OPEN IN VIEWER

Figure 6

Forest plot for general academic ability.

OPEN IN VIEWER

Figure 7

Forest plot for language/literacy. Positive effect sizes favor Montessori education over traditional education.

OPEN IN VIEWER

Figure 8

Forest plot for mathematics. Positive effect sizes favor Montessori education over traditional education.

OPEN IN VIEWER

Figure 9

Forest plot for science. Positive outcomes favor Montessori education over traditional education.

OPEN IN VIEWER

Figure 10

Forest plot for social studies. Positive effect sizes favor Montessori education over traditional education.

Moderator analysis for academic outcomes

Treatment duration and follow-up measurements

Table 8 is a meta-regression table showing Montessori education's relative academic effectiveness over traditional education as a function of treatment duration (i.e., the duration of the Montessori intervention in number of weeks). Within-study effects refer to effects that estimate growth over time on the same set of students within a study. The interpretation of the within-subjects effect (Hedges' g = 0.007, 95% CI [−0.087, 0.100]) is that for every 1 week in Montessori education, the effect size increased by 0.007 standard deviations. Figure 11 is a scatterplot of treatment duration and between-study effect size. Between-study effects estimate growth over time by comparing treatment duration effects between different studies. The interpretation of the between-subjects effect (Hedges' g = 0.002, 95% CI [−0.005, 0.009]) is that for every 1 week in Montessori education, the effect size increased by 0.002 standard deviations.

OPEN IN VIEWER

Table 8

Cluster robust meta-regression for treatment duration (weeks) for academic effect sizes.

Effect	Hedges’ g	95% CI of g	df	p
Intercepta	0.188	(−0.045, 0.420)	9.13	0.101
Within-study effect by week	0.007	(−0.087, 0.100)	1.00	0.538
Between-study effect by week	0.002	(−0.005, 0.009)	2.36	0.353

Note: N of studies = 18, N of effect sizes = 67, assumed ρ = 0.8, I ² = 80.72. Cluster robust estimates with df < 4.00 should be interpreted with caution (Tanner-Smith, Tipton, & Polanin, 2016). Positive effect sizes favor Montessori over traditional education.

Abbreviation: CI, confidence interval.

Reference category/intercept.

OPEN IN VIEWER

Figure 11

Scatterplot of treatment duration and observed effect size for academic outcomes. Positive effect sizes favor Montessori education over traditional education.

While both within-study and between-study estimates indicate a slightly positive linear relationship between effect size and duration in Montessori education, the wide 95% CIs, the dfs less than four, the nonlinearity, and nonconstant variance of the relationship between effect size and duration lead us to conclude that there is too much uncertainty and ambiguity to interpret these treatment duration results with any meaningful degree of certainty. In short, the results for this moderator were inconclusive.

Four studies included in this review used follow-up measurements of academic outcomes after the Montessori intervention period had ended. We present the results of that analysis in Table 9. The results ostensibly show a slight fading effect on academic outcomes the longer a student had been out of Montessori education. Specifically, academic effect sizes decreased by 0.07 standard deviations per year after a student finished Montessori education (Hedges' g = −0.07, 95% CI [−0.16, 0.01]). Because of the low df (1.35) and the nonlinearity and nonconstant variance in the scatterplot, we suggest that readers interpret these results as largely inconclusive.

OPEN IN VIEWER

Table 9

Cluster robust meta-regression for follow-up years for academic effect sizes (between).

Effect	Hedges’ g	95% CI of g	df	p
Intercepta	0.92	(0.15, 1.68)	1.06	0.041
Follow-up years	−0.07	(−0.16, 0.01)	1.35	0.062

Note: N of studies = 4, N of effect sizes = 37, assumed ρ = 0.8, I ² = 31.85. Cluster robust estimates with df < 4.00 should be interpreted with caution (Tanner-Smith, Tipton, & Polanin, 2016). Positive effect sizes favor Montessori over traditional education.

Abbreviation: CI, confidence interval.

Reference category/intercept.

Sensitivity analysis for academic outcomes

A sensitivity analysis examines the degree to which methodological or analytical decisions covary with outcomes (Higgins, 2021). In this review, we conducted several types of sensitivity analyses, the results of which would be too voluminous to detail here in entirety. Therefore, we summarize some of the results of the sensitivity analysis narratively in this paragraph and concentrate on one particularly important sensitivity analysis in the remainder of this section.

In terms of the ρ parameter (i.e., the estimated correlation between dependent effect sizes) used in cluster-robust effect size estimation, the results were consistent regardless of the value of the parameter for ρ (i.e., the estimated correlation between dependent effect sizes) we chose. We conducted leave-one-out analyses for applicable outcomes and also found that the leave-one-out results were consistent with what was reported here.

For main effects analyses, we compared random-effects and cluster-robust synthesis methods. The overall results were consistent in terms of point estimates of effect size; however, as expected, the variance estimates differed slightly between random-effects and cluster-robust models. The random-effects models tended to have lower variance than the cluster-robust models, but we used cluster-robust models nonetheless because we believed that the benefits of taking into account the dependencies between effect sizes outweighed slightly more accurate variance estimates, as discussed in Tanner-Smith (2014).

Fixed-effect models had smaller 95% CIs than cluster-robust or random-effects models, as expected, and were less favorable to Montessori education. We attribute this to the fact that fixed-effect models tend to weight studies with large samples heavily compared to other models and fixed-effect models do not take into account studies with multiple effect sizes (Higgins, 2021). We believe that Culclasure (2018) was overweighted in fixed-effect models because it tended to contribute many effect sizes and had sample sizes in the thousands. As we discuss in the Discussion section, Culclasure (2018), as well as Ansari (2014), lacked consistently high treatment fidelity in the implementation of Montessori education and, thus, these studies might likely suppress the effect size in favor of traditional education.

One sensitivity analysis of particular importance was related to our method of effect size estimation. We used either a method for standardized mean differences, a standardized mean raw score method, or an other method where we calculated an effect size using Wilson (n.d.)'s Practical Meta-Analysis Effect Size Calculator. The results of that sensitivity analysis are shown in Table 10. For academic outcomes, the mean difference in effect sizes between the reference category and other categories was less than 0.04 standard deviations in absolute value and the results of the omnibus Wald test yielded a p value of 0.955, F(2, 2.26) = 0.05. In this analysis the dfs were low and, therefore, these estimates should be interpreted with caution. Nonetheless, with all of the evidence taken together, we believed that it was reasonable to conclude that the method of effect size estimation was not an important moderator of effect size results and, therefore, it is justifiable to aggregate effect sizes across estimation methods as we did in the majority of analyses in this review.

Finally, for those interested in null hypothesis testing, we calculated the Benjamini–Hochberg (Benjamini, 1995; Benjamini, 2005) adjusted α to help readers account for the multiplicity of statistics tests (see Wasserstein, 2019), at least for main effects. A list of those adjustments can be found in the online supplement (Randolph, 2021).

Possible publication bias for academic outcomes

Publication bias is a type of systematic bias that occurs when studies with null or negative effect sizes are withheld from the research record, either because of authors not submitting null or negative results for publication or journal reviewers or editors tending to not recommend studies with null or negative results for publication. To examine publication bias, we first conducted a visual analysis of a variety of funnel plots for all academic outcomes combined (see the four panels in Figure 12). In these funnel plots, the vertical axis represents some measure of sample-size-related variability (inverse sample size, raw sample size, square root of the sample size, log sample size) and the horizontal axis represents the effect size with positive effect sizes favoring Montessori education. Each data point within each funnel represents the correspondence of the standard error and effect size for each effect size. In general, larger studies will have less variance and, therefore, will be located near the top of the funnel whereas small studies will be located near the bottom.

OPEN IN VIEWER

Figure 12

Four sample-size based funnel plot variations for a composite of all academic outcomes combined.

Deviations from a funnel shape may be potential indicators of publication bias. See Higgins (2021) for more information on the visual analysis of funnel plots. One particular deviation to note is when a funnel plot is asymmetrically in favor of the intervention. Asymmetrical funnel plots with a large number of data points outside of the funnel only on the side favoring the intervention may be an indicator of publication bias in favor of the intervention. As a result of too many data points falling outside the funnel, there may be blank areas on the other side of the funnel. This deviation is seen to some degree in many of the main-effect funnel plots presented in Figure 12. There is a preponderance of small studies in favor of Montessori education and an absence of small to medium size studies in favor of traditional education. The result is that the center reference line is also slightly to the right of the couple of studies with many effect sizes with very large sample sizes (e.g., Culclasure, 2018). As we explain in the Discussion section, there are some characteristics of the large sample studies that might attenuate their effect sizes in favor of traditional education. In addition, some of the variability of the handful of extreme outliers on the right can be explained through an analysis of study characteristics. The supplemental information (Randolph, 2021) contains funnel plots for individual academic outcomes.

In addition to a visual examination of funnel plots, we also conducted a trim and fill analysis. Using Duval's (2000) trimandfill algorithm enabled us to determine the number of effect sizes that may have been missing due to publication bias, impute the missing effect sizes to minimize funnel plot asymmetry, and then estimate a new effect size with the imputed effect sizes included.

We provide an example of that procedure here starting in terms of the language/literacy outcome. Figure 13 is a funnel plot in which the black data points represent the 46 observed effect sizes for language/literacy. The 11 data points in white on the lefthand side of the funnel plot represent the effect sizes imputed by the trim and fill algorithm to bring symmetry to the funnel plot as a whole. The value of Hedges' g for language/literacy was 0.24 (95% CI: 0.06, 0.36) for the 46 black data points. After imputing the 11 white data points and pooling them with the original effect sizes, the value of Hedges' g decreased to 0.13 (95% CI: 0.03, 0.23) for the new set of 57 effect sizes (i.e., the 46 original effect sizes and 11 imputed effect sizes). Duval (2000) notes that the trim and fill effect size should not be interpreted as a publication-bias-adjusted effect size. Instead, they recommend that one should use the difference between the imputed and unimputed effect sizes to gauge the degree of potential bias for a given outcome. For example, the 0.11 standard deviation difference between the imputed and unimputed effect size estimates is a rough estimate of the degree to which publication bias may be present in the Montessori research regarding language and literacy.

OPEN IN VIEWER

Figure 13

Funnel plot with trim and fill for language/literacy. The black data points represent observed effect sizes. The white data points represent effect sizes that were imputed using Duval's (2000) trim and fill algorithm to bring symmetry to the funnel plot. Without imputation, Hedges’ g was 0.24 compared to 0.13 when effect sizes were imputed.

Table 11 shows the trim and fill results for the other academic outcomes. Since language/literacy contributed 45 effect sizes when all academic outcomes were combined, it is no surprise that there was statistical evidence of asymmetry for that composite outcome. After imputing 20 left-side effect sizes, the values of Hedges' g for all academic outcomes combined decreased by about 1/10th of a standard deviation--from 0.24 (95% CI: 0.17, 0.29) to 0.14 (95% CI: 0.07, 0.23).

In summary, we found strong evidence for a publication bias effect on the language literacy outcome. We estimate that the publication bias effect for language/literacy is on the order of about 1/10th of a standard deviation in a way that is systematically biased in favor of Montessori education. However, we believe that the degree of systematic bias in favor of Montessori education is not sufficient to negate the overall finding that Montessori education increases language/literacy outcomes when compared to traditional education.

Main effects for nonacademic outcomes

Table 12 and Figure 14 (a histogram of effect sizes for nonacademic outcomes) show the main effects for all nonacademic outcomes combined and individually for the nonacademic outcomes of creativity, executive function, inner experience of school, and social skills. On average, Montessori students performed 0.33 standard deviations higher than traditional students on nonacademic outcomes, 95% CI [0.16, 0.50], with moderate quality of evidence.

OPEN IN VIEWER

Figure 14

Kernel density plot of unweighted values of Hedges’ g for individual nonacademic outcomes.

See Figure 15 for a graphical display of study heterogeneity and effect sizes of academic outcomes from a bootstrap analysis of 1,000,000 samples of our nonacademic effect sizes. This plot shows that the plausible values of the summary effect size for nonacademic outcomes range approximately from 0.10 to 0.35 with high study heterogeneity.

OPEN IN VIEWER

Figure 15

Graphical display of study heterogeneity for all nonacademic outcomes combined. In the GOSH plot above, the X-axis (Overall Estimate) shows the Hedges’ g effect size. Positive effect sizes favor Montessori education over traditional education. The Y-axis represents a measure of study heterogeneity, I ², where higher values represent more study heterogeneity. The black data points are simulated values of the population effect size and heterogeneity.

In terms of the individual four nonacademic outcomes shown in Table 12, there was moderate quality evidence that Montessori education outperformed traditional education. The largest effect sizes were found for executive function (Figure 16) (Hedges' g = 0.36, 95% CI [0.15, 0.58]) and inner experience of school (Figure 17) (Hedges' g = 0.41, 95% CI [0.19, 0.62]). However, the cluster-robust df for inner experience of school (df = 3.79) was slightly under 4.00, indicating that the result should be interpreted with caution (Tanner-Smith, 2016). While there were also small to moderate effects in favor of Montessori education for creativity (Figure 18) (Hedges' g = 0.26, 95% CI [−0.21, 0.74]), and social skills (Figure 19) (Hedges' g = 0.23, 95% CI [−0.02, 0.49]), there was significant variation in the 95% confidence intervals. There was also evidence of high study heterogeneity for nonacademic outcomes as demonstrated by the high values of I ² in Table 10, which ranged from 82.92% for social skills to 66.61% for inner experience of school. Further, while creativity and executive function had moderate quality of evidence, inner experience of school and social skills both had low quality of evidence. See the Summary of findings Table 1 for more information on the certainty of evidence. Overall, there is moderate quality of evidence that Montessori education leads to better performance on nonacademic outcomes than traditional education.

OPEN IN VIEWER

Figure 16

Forest plot for executive function. Positive effect sizes favor Montessori education over traditional education.

OPEN IN VIEWER

Figure 17

Forest plot for inner experience of school. Positive effect sizes favor Montessori education over traditional education.

OPEN IN VIEWER

Figure 18

Forest plot for creativity. Positive effect sizes favor Montessori education over traditional education.

OPEN IN VIEWER

Figure 19

Forest plot for social skills and behavior. Positive effect sizes favor Montessori education over traditional education.

Moderator analysis for nonacademic outcomes

Table 13 shows the meta-regression results for random versus nonrandom assignment for all nonacademic outcomes combined. Similar to the results for nonacademic outcomes, effect sizes from studies with random assignment had an effect size 0.31 standard deviations greater than effect sizes without random assignment; however, there was significant variance in the plausible range of values for this estimate, 95% CI [−0.54, 1.16] and the df < 4.0.

Similar to the effects for academic outcomes, Montessori education had greater impacts than traditional education at all measured grade levels on nonacademic outcomes as shown in Table 14. (There were no included studies with nonacademic effect sizes for high school students). The greatest impacts of Montessori education for nonacademic outcomes were seen at the preschool level (Hedges' g = 0.36, i.e., 0.26 + 0.10 = 0.36). However, a Wald test for an omnibus effect of a grade-level moderator was not statistically significant (p = 0.863). In summary, Montessori education had the greatest impacts over traditional education on nonacademic outcomes at the preschool level; however, there was much statistical uncertainty about grade level as a moderator of the effectiveness of Montessori education.

As Table 15 shows, both private and public Montessori education showed greater impacts on nonacademic outcomes than traditional education. This result was especially strong for private Montessori education (Hedges' g = 0.43, 95% CI [0.18, 0.67]). Private Montessori outperformed public Montessori on nonacademic outcomes. There is statistical uncertainty, however, about the degree to which these results would accurately generalize to the population of students.

Table 16 and Figure 20 show the results of an examination of nonacademic effect sizes as a function of treatment duration. The within-study and between-study effects were contradictory; the within-study effect showed a very slight treatment-duration effect in favor of Montessori education (Hedges' g = 0.001, 95% CI [−0.089, 0.091)] and the between-study effect showed the same treatment-duration effect but in favor of traditional education, (Hedges' g = −0.004, 95% CI [−0.011, 0.003] for nonacademic outcomes. For the same reasons explained in the section on treatment duration for academic outcomes, we regard these results as inconclusive. There were too few studies to do a moderator analysis of follow-up studies for nonacademic outcomes.

OPEN IN VIEWER

Figure 20

Scatterplot of treatment duration (weeks) and observed effect size for nonacademic outcomes. Positive effect sizes favor Montessori education over traditional education.

Sensitivity analysis for nonacademic outcomes

The results of the sensitivity analysis for nonacademic outcomes yielded nearly identical results as for academic outcomes: the difference in results between analytical methods was unremarkable. In summary, the results were consistent regardless of the value of the parameter for ρ that was chosen. The leave-one-out results differed only marginally from the effect sizes reported when all studies were included. For main effects analyses, when comparing random-effects and cluster-robust synthesis methods, the overall results were consistent in terms of point estimates of effect size; however, as expected, the variance estimates differed slightly between random-effects and cluster-robust models. Although not reported here, the fixed-effect models had smaller 95% CIs and were less favorable of Montessori education, likely due to implementation in the large sample studies that will be discussed later.

In terms of the differences between effect sizes rendered using different estimation methods for nonacademic outcomes (see Table 17), there was a large effect size difference between the standardized mean raw score change method and the other method (Hedges' g = −0.45, 95% CI [−2.36, 1.45]) and between the standardized mean difference method and the other method (Hedges' g = −0.54, 95% CI [−2.11, 1.01]), as shown in Table 13. However, the mean difference between the two standardized methods, which consisted of the majority of effect sizes, was only 0.06 standard deviations and the Wald test for omnibus differences yielded a p value of 0.382, F(2, 1.57) = 1.57. This suggests we are justified in synthesizing results regardless of the method we used to calculate a study's effect size.

Possible publication bias for nonacademic outcomes

Figure 21 is a four-panel funnel plot of all nonacademic outcomes combined. The same asymmetry of small studies in favor of Montessori education that was seen for academic outcomes is also seen here for nonacademic outcomes. Similarly, we believe that some of the variance can be explained through an analysis of study characteristics of the extreme outlying studies and of the studies with a very large sample size, as explained in the Discussion. We conclude that there is some slight asymmetry in small studies in favor of Montessori education but it is unclear whether publication bias is the cause. Asymmetry alone is not sufficient evidence for publication bias (Higgins, 2021). Overall, because of the small degree of symmetry and the analytical methods used, we conclude that the asymmetry does not change the substantive conclusion that Montessori education is favorable for most nonacademic outcomes. Conservatively, we believe that the range of plausible values of the aggregate effect of Montessori education on nonacademic outcomes is represented well in the GOSH plot shown in Figure 15.

OPEN IN VIEWER

Figure 21

Four sample-size based funnel plot variations for nonacademic outcomes.

Table 18 shows the results with a trim and fill analysis to help interpret the degree of publication bias. Similar to the trim and fill analysis conducted for academic outcomes, we estimate that publication bias favors Montessori by about 1/10th of a standard deviation.

Study design: Random versus nonrandom assignment

The major obstacle to drawing conclusions about outcomes from freely chosen school programs is selection bias. One can control for family income, ethnicity, and other factors, but the possibility that families who choose Montessori differ in some other way that accounts for the results remains. Ensuring equality on key variables at pretest is helpful since presumably the family factors that cause a child to be higher on some outcome variable would be present at the outset as well. Controlling for the level of that variable at pretest is also helpful, but if the family produces an exponential growth pattern on that variable, it is uncontrolled. Because family effects that might positively affect child outcomes are confounded with Montessori enrollment, one would expect effects from nonrandom experimental designs to be greater than for random assignment studies. Interestingly, here this was not the case. Rather, results were considerably stronger for studies using random assignment for academic effects, whereby random assignment raised the effect size of 0.19 found with studies that used pretests or control variables to ensure baseline equivalence, to almost 1/3 of a standard deviation (i.e., 0.31). For nonacademic effects, the nonrandom assignment effect size of 0.28 more than doubled, adding 0.31 standard deviations. Just six studies had random assignment; three used data from children randomly assigned to a Montessori intervention at age four through 10th grade, with results published in three included papers (earlier reporting on that same study was not included because effect sizes could not be calculated from the information provided therein), and found strong sleeper effects for children previously enrolled in a Montessori Head Start, especially for boys. Two studies involved lotteries at oversubscribed American public schools. The sixth study was of 60 children enrolled in preschools in a city in Iran; it randomly assigned half the children to a targeted Montessori intervention. It contributed just two effects and the report is not sufficiently clear to allow speculation regarding the reason for its strong effects.

Focusing on the other studies, we speculate that their outcome effects stem from study timing in one case (representing three papers), and implementation of Montessori in the other two. Regarding the former, it might be that even weak Montessori implementation has long-term effects because philosophical elements like free choice and order can exist even when the structural aspects of implementation are weak; these philosophical elements might not always manifest in differences in the near term, but could manifest later. The Miller studies, which contributed many effect sizes to the random effect, did not have particularly strong Montessori implementation (the original Miller, 1975 study reported an implementation score of 6.5/10), but it did provide for free choice and other philosophical elements, and this early experience at four might have led to long-term outcome differences relative to the counterfactual (which was a traditional free play school). Another study of long-term academic outcomes (not included here because it lacked appropriate controls) also found better academic performance for Montessori students (with unclear program implementation) even years after they had left the program (Dohrmann, 2007). Other studies suggest that without regard to implementation, Montessori predicts better nonacademic outcomes (LeBoeuf, 2022; Lillard, 2021).

The two Lillard random lottery studies looked at the immediate effects of Montessori, but the three oversubscribed public schools in the two studies were particularly strong examples of Montessori implementation. Although oversubscription is not itself a clear indicator of school quality (Tuttle, 2012; Weiland, 2020), here the oversubscribed schools were all AMI (Association Montessori Internationale) recognized Montessori schools. AMI schools adhere to the structural elements (i.e., they hire AMI-trained teachers, which is an intensive 9-month, standardized training with a highly prepared and vetted trainer; they have the specific 3-year age spans, scarce adults and high ratios; and the long work periods and full set of Montessori materials) and they also implement the philosophical elements well. By contrast, teachers trained by the other major training organization, the American Montessori Society (AMS), founded to “Americanize” Montessori education (Rambusch, 1992), are relatively more supportive of conventional American education practices like tests, due dates, worksheets, and whole-class activities (Daoust, 2018, April), which might dilute the immediate effects of the Montessori program as compared to traditional education. Two studies specifically examined the influence of implementation on outcomes and found that the more classic implementation espoused by AMI is associated with better outcomes than supplemented implementations (Lillard, 2012; Lillard, 2016).

In sum, we speculate that the randomized trials had better outcomes in this meta-analysis because sleeper effects on outcomes occur in response to philosophical aspects of Montessori that exist even when structural implementation is weak, and because two of the random studies, while looking at immediate outcomes, had especially high fidelity Montessori implementation.

To shed further light on why random assignment had an unexpectedly stronger effect than nonrandom assignment in this meta-analysis, more research is needed using random assignment while also coding for Montessori implementation. We would hypothesize that among studies using random assignment, those using a more classic Montessori implementation would have stronger effects.

Age level

Academic effects were strongest for children in Elementary school, at 0.36. As is typical, effects were smaller across middle and high school; effect sizes for academic achievement across a school year decline fairly steadily from K to 12 (Hill, 2008). However, for Montessori, effects at preschool (0.20) were also smaller than effects at Elementary school (0.36). Yet the preschool effects are still notable. For example, one random-assignment study of Head Start, with children enrolled either at age three or age four, found that 13 of 22 effects on language, literacy, and math were significant, and of those that achieved significance, the average effect size was 0.18 SD (Barnett, 2011), similar to our preschool effect which was not limited to those that achieved significance. The greatest gains for most children are typically seen from Kindergarten to first grade (Hill, 2008) when children undergo the famous “5 to 7 shift” (Sameroff, 1996) with biological changes augmenting environmentally-induced ones. In Elementary school, the average growth across each school year is 0.44 (Lipsey, 2012), thus Montessori education may add as much as 80% of the gains expected in an entire school year to children's achievement. Although effect sizes were calculated for middle and high school and were considerably smaller, very few studies contributed to effect sizes at those levels giving us less confidence in those effects. Nonacademic effects were strongest for young children, which may indicate that at those ages children's executive function, creativity, and so on are most malleable. There are few developmental studies of these abilities, thus we were unable to view these results comparatively.

Public versus private Montessori

As Table 7 showed, both public and private Montessori educational interventions had better results than traditional education on aggregated academic outcomes. Public Montessori schools were shown to have an effect size 0.13 standard deviations lower than private Montessori schools on aggregate academic outcomes. This difference is equivalent to 29% of the average growth expected in a school year for elementary students (Lipsey, 2012). Nonetheless, the 95% CIs showed that the range of plausible values for this estimate varied from public Montessori education's performing 0.36 standard deviations lower than private Montessori education, which is equivalent to 82% of the average yearly academic growth expected in school in elementary education, to public Montessori education's performing 0.12 standard deviations above private Montessori education, which is equivalent to 27% of the average academic growth expected in elementary education. Therefore, we conclude that private Montessori education is likely to achieve academic results as good as or somewhat better than public Montessori education.

As Table 15 showed, both public and private Montessori educational interventions had better results than traditional education on aggregated nonacademic outcomes. Private Montessori had an effect size of 0.43 for aggregated nonacademic outcomes; public Montessori had an effect size of 0.26 standard deviations less than private Montessori. The effect size differences between public Montessori and private Montessori were twice as great for nonacademic outcomes, which had a 0.26 effect size difference in favor of private Montessori, as academic outcomes, which had a 0.13 effect size difference in favor of private Montessori. Because there are few longitudinal studies of the nonacademic outcomes reported here, it is not possible to view these effects in terms of expected yearly growth.

In summary, we conclude that there is preliminary evidence that private Montessori education leads to moderately greater performance in both academic and nonacademic outcomes than public Montessori education. We can also conclude that there is preliminary evidence that this difference in improvement for private Montessori and public Montessori is likely to be greater for nonacademic outcomes than academic outcomes. We argue that the evidence is preliminary because of the statistical imprecision of our estimates.

The public–private difference might be due to implementation. Public schools are required to have children take state exams that are designed for traditional school programs; Montessori schools need to adjust their program to prepare for those tests, and this by necessity dilutes Montessori implementation at public schools. Public schools are also required to follow regulations about recess breaks, special classes in art and sports, add-in curricula, and teacher-child ratios and class sizes that compromise the integrity of the Montessori program. Private schools also sometimes make such compromises, but they are more free to determine their programs.