Abstract
Linked article:
Background
The Problem, Condition, or Issue
In the early 1900s, Maria Montessori characterized schools as places of “enforced mobility, inadequate light, [poor] ventilation, and rote methodology” (Turner, 1992, p. 18). Given this confined topography of the educational landscape, Montessori developed a holistic approach to schooling—one that was meant to liberate children via a developmentally appropriate approach (Montessori, 1964). This rounded approach involved an intentional interest in improving the quality of students’ lives. The holistic approach, which included “opportunities to solve problems on their own, question assumptions, and reason their way to solutions” (Diamond, 2010, p. 781), was facilitated through a curriculum and environment that encouraged engagement and the cultivation of students' innate academic and social abilities.
The Montessori method is an attractive alternative to traditional education for many parents in the United States of America and abroad (Lillard & Else-Quest, 2006). There are between 4,000 and 5,000 private Montessori programs in the United States, and thousands more internationally (American Montessori Society, 2016; Lillard & Else-Quest, 2006, North American Montessori Teachers' Association, 2016; USA Montessori Census, 2016). Apart from private Montessori schools, there are at least 500 public Montessori schools throughout the United States (National Center for Montessori in the Public Sector, 2016).
Numerous qualitative narratives espousing the benefits of Montessori education to enhance executive functioning (e.g., working memory, self-regulation, and reasoning abilities), academic achievement, and social competence are readily available. Despite the existence of these narratives, hardly any empirical consensus regarding the actual utility of the Montessori approach can be found (Diamond, 2010).
The purpose of this review is to investigate, via a quantitative meta-analysis, the hypothesis that the Montessori method is at least as effective as traditional education in affecting academic and social outcomes for children. The proposed meta-analysis is completed with the intention to help the public, as well as the research community, make more informed and empirically sound decisions regarding Montessori education by collecting, codifying, synthesizing, and disseminating the current empirical research.
Montessori Education Defined
To appropriately communicate the purpose, meaning, and utility of Montessori education across a variety of sociocultural and economic contexts, the topic of Montessori education must be examined through the lenses of the meaningful subdomains into which it subsists. These domains include the following three frameworks: (a) the historical background, (b) a conceptual analysis of how schools most often employ the intervention, and (c) the theoretical underpinnings of Montessori as a philosophy and approach to curriculum.
Historical Considerations
According to the American Montessori Society (AMS, 2016) and Turner (1992), the Montessori method is not so much a “curricular model” as it is a “philosophy” (AMS, 2016; Turner, 1992). The Montessori model is based on the interactive dynamics of three major components—the child, the adult/teacher, and the environment (see Figure 1), as well as a philosophy that views children as naturally inquisitive and capable initiators who are eager to understand the world around them.
The Montessori triangle.
To nurture and optimize children's understanding of their environment and the world, Montessori believed that the physical environment must be an environment in which learners could be supported and could feel accepted as they progressed through their innately unique processes of learning. According to Diamond (2010), the absence of this positive classroom environment minimizes children's sense of safety, security, and acceptance, which significantly hinders the likelihood that the child will feel confident enough to take risks to learn. To accomplish this goal, Montessori designed the curriculum, and the learning environment in which the curriculum is situated. It was based on the idea that students be granted doable, or developmentally appropriate opportunities, or “invitations” to support their natural curiosity through movement in an organized environment (Diamond, 2010; Ervin, Wash, & Mecca, 2010; Faulkenberry & Faulkenberry, 2006; Ultanir, 2012).
Lillard (2005) and Jones (2005) outlined eight Montessorian principles that the organized learning environment must uphold to accomplish the goal of nurturing and optimizing student learning. The environment must be conducive to: Movement and cognition (i.e., movement enhances thinking and learning), Student Choice, Student Interest, Avoidance of extrinsic rewards, Collaborative and cooperative learning, Context-dependent learning, Teacher models and student methods, and, Order in the environment.
These eight principles of Montessori education are manifested in the physical and social environment of the Montessori classroom. For example, concerning collaborative and cooperative arrangements conducive to learning, older students often work with younger children to learn from what the older children have already mastered. This multi-age grouping (versus the strict age grouping seen in U.S. schools) inspires the younger children through intrinsic motivation (i.e., desire to understand and engage), to imitate, assimilate, and accommodate the academic or social skill authentically and with mastery (Turner, 1992). Nunnery, Chappell, and Arnold (2011) provided yet another example of how these eight principles are evident in the Montessori classroom. When entering a Montessori classroom, it is physically and audibly apparent that Montessori teachers bolster an awareness of situated problems within daily life such as housekeeping, art, cooking, gardening, or experimenting. Montessori teachers also facilitate student-driven creative approaches to solving these problems as well as determining the lessons that can be gleaned from them (Nunnery, Chappell, & Arnold, 2011; Ultanir, 2006).
It is evident from the pedagogical examples provided here and in the research literature regarding the characteristics of the Montessori classroom that the geographical location is immaterial. The core of the Montessori educational environment is built around the three interdependent components of: (a) the teacher, (b) the child, and the (c) educational environment (see Figure 1).
Montessori teacher training and accreditation programs (e.g., Association Montessori International, American Montessori Society, Pan American Montessori Society, International Association of Progressive Montessori, etc.) vary internationally. Therefore, there will be slight variations in the Montessorian curricula in regards to the physical setting, type of resources available, and the pedagogical practices employed. The following section works to delineate the nuanced variations cited within the research literature.
Conceptual Frameworks: The Intervention
Before disaggregating the differences amongst Montessori programs, the reader should be aware of the fundamental commonalities often found within Montessori classrooms. A characteristic feature of all Montessori programs is the consistent age groupings of Montessori attendees—specifically for primary children. According to Lillard (2013):
Montessori classrooms ideally contain age groupings spanning three years: Infant to three years old, three to six, six to nine, and nine to twelve. [Age-appropriate] working materials, kept on shelves and freely available to the children, are organized into topics such as language, math, and so on. The materials are designed so that if children make mistakes, they can see and correct them without close teacher supervision or intervention. Areas of the curriculum are tightly interconnected (Lillard, 2013, p. 159).
For children between the ages of three to six, the primary classroom includes five areas of self-directed exploration. These include the: (a) cultural area, which includes geography, history, the sciences, and an exploration of human activity (e.g., music, dress, nutrition, etc.); (b) “practical-life materials, mirroring the activities of the culture, such as care of self and the environment; (c) manners and social behavior; (d) sensorial materials, reflecting the qualities and facts about the world; and (e) mathematics and language materials” (Lillard, 1996, p. 34). The materials are arranged attractively and sequenced from simplest to most complex, and the children are encouraged to engage in critical exploratory work within these five domains. This inherent order in the arrangement of the environment aids the children in better understanding the myriad impressions that they receive through their senses, and in the development of a rich vocabulary for describing these experiences.
For children between the ages of six to nine and nine to twelve, Montessori classroom and process of education is similar for both the upper and lower elementary groups, with the main difference being the maturity of the latter group in how they interact with the environment (Lillard, 1996). Lillard noted that “nine- to twelve-year-olds go into much further detail in their studies and their research, and their ‘going out’ activities reflect a greater depth of knowledge and intensity of interests” (p. 115). Montessori asserted another commonality between these age groups: children between the ages of 6 to 12 years enjoyed a period of physiological and psychological stability marked by a dramatic expansion of the imagination (Montessori, 1912/1964). Lillard describes the characteristics of the upper and lower elementary classrooms below:
There are special materials displayed on shelves in an organized manner by sections: mathematics, geography, science, art, music language and so forth… . The elementary classroom can best be described as a workplace. Although the noise level may be somewhat higher than in the primary classroom, the orderliness of the environment leads to concentration and careful effort (pp. 78-79).
In addition to these characteristic features of Montessori programs which maintains the historical vision of the Montessori method, many Montessori programs follow some adaptation of Turner's (1992) learning cycle model for Montessori education. Although the model is complex and rooted in the triangular historical model of the child, the adult, and the environment, the learning cycle model is fundamentally embedded in and provides a springboard for how experiential learning can occur (Turner, 1992; see Figure 2).
The natural learning cycle.
According to Turner (1992), the natural learning cycle includes three phases. The first of which requires that the student observes a demonstration given by a teacher or another student. Next, the student is given an opportunity to participate in the demonstration, and then allowed to practice. The amount of time needed to assimilate the relationship between the stimulus and phenomenon varies per child, but once it occurs, it is during this period that the student experiences a moment of conscientization, or “Eureka moment.” Following this moment of revelation comes the student's ability to perform effectively. It is when the student can perform effectively—and can pass on this knowledge to other students (through demonstration)—that the cycle of learning is considered concluded.
Many Montessori institutions (and programs) that employ the Montessori method maintain fidelity to the archetypal philosophy of the Montessori approach (i.e., the students’ learning cycle is shaped via a natural curiosity about the learners' environment and social competencies). Nevertheless, theoreticians and Montessori directors do not always agree on certain nuances of the approach. For example, empirical studies conducted by Bagby, Barnard-Brak, Sulak, Jones, and Walter (2011), Lillard (2012), and McDurham (2011) evidence variations in: Traditional age groupings of students (i.e., not in 3-year increments), Variation in student-teacher ratios, Absences or abundances of Montessorian resource materials, Teacher oriented classroom (versus the student-centered classroom), and Presence of thematic or themed school (especially in the public sector).
Variations in the perceived intention, and therefore the methodology of Maria Montessori's writings, have probably been the reasons for the previously mentioned inconsistencies across programs and even accrediting agencies. These variations in methodology and the range of empirically purported benefits given these variations have probably yielded conflicting and inconclusive results regarding the efficacy of Montessori education within the research literature.
Theoretical Underpinnings: A Social Constructivist Approach to Teaching and Learning
Although empirically challenging, the variation in approaches to Montessori education is what ties the Montessori method to constructivist theories of learning. In the Tylerian approach to pedagogy, the teacher is the bestower of knowledge whereas, in the Montessorian approach, the student serves as the principal constructor of knowledge (Loeffler, 1992). Both Montessori and constructivism posit that truth and knowledge are a working hypothesis, not imposed from outside people. Both approaches are based on people's beliefs and experiences in situations, and hold that educational curricula must address the cognitive, emotional, social, and physical needs of the whole child (Diamond, 2010; Hyslop-Margison & Strobel, 2008; Loeffler, 1992; Schunk, 2012). Under both Montessori and constructivist methods, the role of the teacher is to observe carefully and facilitate students’ academic, social, and affective development in realistic, complex, and relevant learning environments by focusing on the needs of the “whole” child (Diamond, 2010; Driscoll, 2005). Similar to the historical conceptions of Montessori education, constructivism does not subsist as a theory, but as an epistemology, or way of believing about the nature of knowing (Schunk, 2012; Wu & Tsai, 2005).
With this viewpoint, the goal of constructivists, regarding the praxis of teaching and learning, is to integrate materials and experiences into the learning environment that encourage dialectical constructivism, or knowledge gained from interactions between persons and/or their environments (Meece, 2002; Schunk, 2012). Vygotsky's theory of social constructivism remarkably mirrors those collaborative and cooperative experiences echoed throughout Montessori axiology. For example, Vygotsky's social constructivism contends that: Social interactions are critical; knowledge is co-constructed between two or more people; Self-regulation is developed through internalization of actions and mental operations that occur in social interactions; Human development occurs through the cultural transmission of tools (e.g., language and symbols); Language is the most critical tool [and it] develops from social speech, private speech, to covert speech; and The zone of proximal development is the difference between what children can do on their own and what they can do with the assistance from others (Schunk, 2012, p. 243).
Each of these contentions aligns with one or more of the eight principles of Montessori education (e.g., collaboration and cooperative learning, context-dependent situations, intrinsic versus extrinsic motivation). The contentions correspond to the Montessori triangle which conceptualizes how interactions between the child, the adult, and the environment facilitate learning. It also provides an approach to nurturing and optimizing the natural learning cycle by allowing students to communicate their observations and moments of conscientization during the learning process.
In addition to comparisons amenable to Vygotsky's social constructivism is Piaget's theory of cognitive constructivism. According to this theory, cognitive development develops prior to our ability to socially interact as well as construct and does so through four domains: (a) biological maturation, (b) experience with the physical environment, (c) experience with the social environment, and through these three domains, mature to the fourth—equilibration (Piaget, 1969; Schunk, 2012). Equilibration refers to the innate desire for one's internal and external environments to balance (Piaget, 1969). According to Piaget, this inner desire to create a cognitive correspondence with the external world is what drives the ability and the interest to assimilate and/or accommodate new information. It is this interest in and ability to assimilate and accommodate cognitive disagreement that most closely mirrors the purpose of the Montessori principles (e.g., intrinsic motivation, environmental choice and interest, and cognition through movement) and the cycle of learning (e.g., Turner's Eureka moment). It is also this conception of cognitive constructivism that provides the theoretical rationale behind teachers' introducing realistic, complex, and relevant cognitive conflicts for students to creatively problem solve.
However, despite these fundamental parallels between the nature and purpose of learning, as similarly defined by the Montessori curriculum and theories related to constructivism, there exist some differences. One key example exists between the role of play in the classroom. Montessori vehemently disagreed that the nature and orientation of students’ self-directed learning coincided with Piaget's theory of meaningful play (Lillard, 2013). Montessori characterized the activities that children took up during and while in their learning environment as: “work.” Cognitive constructivists such as Piaget would argue that not only is this self-direction a type of play, but also that play also serves a prerequisite to higher levels of executive functioning as the child matures (Piaget, 1969). Brooks and Brooks (1999) offered yet another basis of disagreement between Montessorians and constructivists, via their description of the dynamics of the constructivist classroom. The constructivist classroom is one in which the teacher is principally concerned with making students' life experiences relate to predetermined standards and learning objectives. In a Montessori classroom, the teacher serves as the conscientious facilitator of knowledge as the destination of the learning cycle is not predetermined. The constructivist approach to the learning cycle requires that teachers “pose problems of emerging relevance”; the Montessorian method allows problems to unfold authentically (Brooks & Brooks, 1999, ix). Constructivism traditionally exists within the behaviorally oriented, or traditional classroom, in which personal modifications to the theory abound. As such, the theory of constructivism resides in classrooms as an instructional approach as opposed to a way of learning and knowing.
Though, while there are some differences between the perspectives and aims of Montessori and constructivist methods, the authors believe these differences to be cursory and not fundamentally incongruous. Moreover, due to the epistemological and axiological foundations of both the Montessori approach and constructivist ideology, in conjunction with the practical pedagogical applications of constructivist theories, the authors contend that the Montessori method, and therefore, Montessori education, is most closely aligned to and underpinned by the constructivist learning theory.
Why it is Important to Conduct this Review
As previously described in the section elucidating the paucity of Montessori meta-analyses, there have been several narrative reviews related to the quality of Montessori education for academic and social outcomes for primary children (e.g., Boehnlein, 1988, 2001; Jones, 2005; Murray, 2010). However, to our knowledge, there has not been a recent and comprehensive meta-analysis of Montessori education or a meta-analysis done with the intention of describing the purported effects of Montessori education on these same outcomes.
Murray's (2010) review of early (1960s and 1970s) and contemporary (2000s to present) Montessori research revealed that that the recent studies that had employed improved statistical methods demonstrated that the results were favorable to Montessori Education. Unfortunately, Murray's review provides an outdated depiction of the current state and value of Montessori education. Jones (2005) reviewed the Montessori education outcomes research in three areas: (1) effects of the Montessori approach to at-risk students, (2) the effects of the Montessori method on exceptional learners including learning disabled, developmentally delayed, and gifted/talented, and (3) comparative analyses of traditional schooling versus Montessori in student achievement and social development. Although thorough, Jones failed to take a systematic approach to searching the literature and did not quantitatively synthesize the research data. This result causes interested stakeholders to surmise the alleged benefits (or liabilities) of the Montessori method. Boehnlein (1988, 1990) reviewed the Montessori education research that may be of interest to public schools, summarizing the results as follows: Early research gives evidence that the Montessori method and environment is beneficial to low and middle SES children. Current research corroborates the early findings, in particular, the importance of Montessori preschool experience. Of specific importance for best results long-term are the full three-year preschool program, trained Montessori teachers, and multi-age grouping. (p. 476)
Similar to the other reviews, the Boehnlein reviews, while thorough, were narrative reviews without a systematic search strategy or quantitative synthesis. In summary, the current narrative reviews of Montessori education lack the systematic quality and rigor afforded by a Campbell Collaboration review.
The authors have located one previous high-quality meta-analysis of Montessori research by Borman, Hewes, Overman, and Brown (2003). In their review, Borman et al., (2003), reviewed nearly 30 comprehensive school reform programs and concluded that Montessori education serves as a reform with “promising evidence of effectiveness,” d = 0.27 [95% CIs 0.19, 0.35], p < .01. Other school reform programs that were also classified as having promising evidence of effectiveness were America's Choice, Atlas Communities, Paideia, and the Learning Network. However, the Borman review only provided outcomes based on national standardized tests of academic achievement.
The proposed meta-analysis will help the public and the research community understand the qualitative and quantitative outcomes of Montessori education. By our collecting, codifying, and synthesizing the empirical research on Montessori education parents, teachers, and policymakers will better able to make informed decisions about Montessori methods and programs.
Objectives
The primary objective of this review will be to examine the effectiveness of Montessori education in improving academic and behavioral outcomes compared to traditional public school education. The secondary outcomes will be to determine what factors moderate the reported effectiveness of Montessori education. Possible moderators include the age of students, setting, type of teacher training, student characteristics, study quality, study design, among others.
Methodology
Criteria for including and excluding studies
Types of study designs
We will include studies that used group experimental and/or quasi-experimental research designs. We will include quasi-experimental designs because we believe there will be very few studies that use random assignment in this context. Studies that used correlational, quantitative descriptive or qualitative designs will be excluded. We will not include regression-discontinuity or single-participant designs because of the complexity and debate over analyzing their effect sizes.
We will exclude experimental/quasi-experimental studies if they do not meet the following What Works Clearinghouse (n.d.) study quality standards. For experimental and quasi-experimental designs, those standards are listed below: Group membership was determined through a random process. or Equivalence is established at the baseline for the groups in the analytic sample.
In other words, we will exclude experimental or quasi-experimental designs that did not use a control group or used a control group but did not establish equivalence at baseline (e.g., through a pretest, matching, or statistical adjustment on important characteristics related to the outcome of interest).
We will exclude studies in which the author did not report enough information to compute means, SDs, and n for each group for continuous outcome variables. We will exclude studies that do not report events and total events (e.g., number who passed an academic achievement test) for dichotomous outcomes. For studies published since 2000, we will attempt to contact authors to get this information if it is not reported in the study. Studies excluded because there is not information to compute the relevant effect size will be narratively described.
We will include studies in which the participants were in preschool, elementary, middle school/junior high school, and high school. Although high school studies are unlikely, we will include them to be comprehensive. We will code for demographic characteristics of participants such as gender, poverty, at-risk status, and disability.
Types of interventions
The intervention will be defined broadly as Montessori education. We will operationalize this as a program that claims to use the Montessori method. We will not include studies which use other types of nontraditional educational interventions (e.g., the Waldorf method) unless Montessori education is a separate measurable intervention. We will exclude studies that do not use traditional public education as the comparison group.
Types of outcome measures
There will be two types of outcome measures: academic and social/behavioral measures. A study will be excluded if does not deal with an academic or behavioral outcome. The academic outcomes will be standardized or teacher-made measures of academic achievement. If possible, we plan to examine academic outcomes by subject matter, especially in core subjects such as math and English language arts/reading. The primary outcomes are the academic outcomes.
There is a large variety of social and behavioral outcomes reported in the Montessori research. For example, in a widely known Montessori study, the authors measured eight different social/behavioral outcomes such as executive function, social cognition, refers to social justice, positive shared play, ambiguous rough play, creative story, positive social strategies, and sense of school as a community (Lillard & Else-Quest, 2006). Because of the wide range of social/behavioral outcomes, we propose to use an emergent approach to arrive at the measures we will ultimately use. The secondary outcomes are the behavioral outcomes.
Moderators
Duration of follow-up
We estimate that the duration of follow-up from pretest to posttest will be one academic semester. In other cases, it may be mid-semester or one academic unit. We will code duration time as study characteristic because we expect it to vary across studies.
Types of settings
In the control condition, we will only include studies in the traditional public education setting or private non-Montessori pre-school setting. In the experimental condition, we will include studies that are in public and private Montessori education settings. Other demographic information that we will collect about setting includes the geographic location of the experiment and the city setting (e.g., urban, suburban, rural).
Search strategy
We will implement a comprehensive search strategy in an attempt to retrieve all experimental and quasi-experimental studies relevant to the review that adhere to the inclusion and exclusion criteria. We will examine both published and unpublished literature. Articles and gray literature will be gathered using online databases that cover education, sociology, and psychology and recommendations from experts in the field of Montessori education. The search will be limited to articles written in English. We will employ both free-text and controlled vocabulary terms in the searches. All permutations of search terms will be used during the search process. We will complement our search with a thorough examination of reference lists of relevant retrieved studies and will contact experts in the field to identify any ongoing or unpublished studies.
Databases
A comprehensive database search will include the following online subscription databases: ProQuest Education Journals EBSCO Education Full-Text ProQuest Dissertations & Theses ERIC EBSCO PsycINFO EBSCO Professional Development Collection EBSCO Teacher Reference Center EBSCO Academic Search Complete EBSCO Sociological Collection EBSCO SocINDEX with Full Text ProQuest Research Library ProQuest Social Sciences Journals JSTOR Social Sciences Citation Index Arts & Humanities Citation Index
Prior reviews and reference lists
We will examine previously collected reviews for inclusion and search the reference lists of retrieved studies to expand the search.
Hand Search
Hand searches will be conducted with the latest available print journals that have yet to be electronically indexed.
Gray Literature
Conference proceedings will be reviewed to find relevant unpublished studies. Government websites will be searched for relevant studies, reports, and citations. Sources for gray literature to be searched include: Society for Research on Educational Effectiveness American Educational Research Association OpenGrey Social Science Research Network Institute of Education Sciences Association Montessori Internationale Association Montessori Internationale / USA American Montessori Society National Center for Montessori in the Public Sector Montessori Educational Programs International Pan American Montessori Society
Internet Search
Google, Google Scholar, and Bing will be used to search the open web in an attempt to fill in any gaps left after searching the specialized sources. Results will be analyzed until a saturation point has been reached, (i.e., until further searching leads to no new articles for inclusion).
Contacting Other Researchers
Authors of prior studies and other experts in the Montessori method will be contacted to obtain unpublished research or to get further clarification on published studies.
Search Methods
During the search process, we will utilize phrase searching and truncation methods to find all variations of relevant search terms. Database thesauri, when available, will be used to find controlled vocabulary descriptors and related descriptors which will be integrated into search iterations.
Search strategies will be tailored to the unique controlled vocabularies of each database and will be used in conjunction with free text search terms. An example of concepts to be searched are as follows:
Description of methods used in primary research
We anticipate that most studies will employ quasi-experimental research designs. Of the few pilot studies we have reviewed, most used the nonrandomized pretest-posttest with control group design (Shadish, Cook, & Campbell, 2002). One typical study was conducted by Manner (1999), in which the author compared third-grade Montessori students with third-grade traditional public school students on Stanford Achievement test scores in reading and mathematics. The author used a repeated-measures design where third-grade students in the same county and similar schools were matched on baseline Stanford Achievement Tests and where students were selected into the Montessori condition through a lottery. There were 30 matched cases for the match outcome and 37 matched cases for the achievement outcome. The students were tracked for three years in terms of the Stanford Achievement test results. The author used a repeated measures analysis with posthoc tests to analyze the results for each outcome. In terms of risk of bias, we determined that there was a risk of reporting bias because the author reported results differently for different outcomes; she concentrated on statistically significant outcomes.
Criteria for determination of independent findings
In the case of multiple outcomes within a study, we will report the outcomes separately when possible (e.g., reporting language arts and reading results separately). If sample sizes are sufficient (N > 15 studies) we will conduct multi-level meta-analysis using the metafor package for R (Viechtbauer, 2010) and the methods in Konstantopolous (2011) when we have multiple outcomes within a study. If the sample sizes are less than or equal to 15, and we have multiple measures of the same outcome (e.g., two different measures of reading achievement), we will select a single outcome based on a “pre-defined hierarchy of outcomes” to reduce selection bias as suggested in the MEC2IR (2014) Standards. The elements of the hierarchy are that (a) we will first choose nationally normed standardized achievement tests over teacher- or researcher-made tests. If element (a) is insufficient to make a decision about which of multiple measures of an outcome to include, (b) we will then choose the measure with the greatest reliability and validity coefficients. If element (b) is insufficient (e.g., there is a tie or no reliability or validity information is available) we will then choose the measure randomly. If there are multiple groups, only the traditional and Montessori groups will be compared. We will collate reports of the same study. If multiple time points are measured, the baseline and the final measurement during the intervention period will be used. If that is not possible, results from the aggregate mean and SD will be used.
Details of study coding categories
Our coding sheet is divided into the following categories: publication characteristics, setting, outcomes, participants, research design (including an indicator of methodological quality and risk of bias), statistical analysis of the study, and characteristics of effect size. The research design, statistical analysis of the study, and characteristics of effect size were based on the Campbell Collaboration Methods Policy Briefs. We did not include the risk of bias variables related to blinding because blinding is not usually possible in educational studies. We will assume a high risk of blinding bias for all studies; therefore, we do not have the risk of bias codes related to blinding. We expect quasi-experimental studies, so we intend to measure what methods and confounding variables the study authors controlled for as suggested. The list below indicates the variables that are planned to be used in the coding sheet. The appendix to this protocol provides more specific information on the variables used in the coding sheet. Publication Characteristics Short title of study Year of study Publication format (e.g., article, book, dissertation, etc.) Setting Geographical setting (e.g., North America, South America, etc.) City setting (e.g., rural, urban, etc.) Control school type (e.g., private, public, charter, etc.) Treatment school type (e.g., private, public, charter, etc.) Outcomes Subject matter (e.g., mathematics, English/language arts, etc.) Subject matter other (e.g., specify if other subject matter) Academic achievement measure (e.g., standardized test, teacher/researcher made test, etc.) Academic achievement measure (write in what measure was used) Social/behavioral measure (write in the measure) Quality of social/behavioral measure (e.g., externally made valid and reliable measure, researcher-made measures w/o validity or reliability information, etc.) Type of social/behavioral measure (we will use an emergent approach to develop categories. Participants Preschool participants Elementary participants Middle school/junior high school participants High school participants Risk of Bias/Research Design Random Assignment Equivalence established at baseline Type of design Equivalence established through matching/blocking Equivalence established through statistical control variables (if yes,) Pretest related to outcome variable Gender of student At-risk status of student Disability status of student SES Race/ethnicity Age of student Setting Other (write in other variables controlled for) Attrition bias Random sequence bias Reporting bias (Write in suspicions about other types of bias in notes). Statistical Analysis of Study For continuous outcomes Mean, standard deviation, and n size of control group Mean, standard deviation, and n size of treatment group For dichotomous outcomes Number of events and number of possible events for control group Number of events and number of possible events for treatment group Other effect size information (write-in) Effect sizes based on raw or statistically adjusted data Significance level for effect size Direction of finding favors Montessori Amount of missing data in percent Treatment interval – Length of the intervention in weeks Follow-up intervals – write a description of how many follow-ups and their spacing over time. Reliability (write in the reliability of the measure and what type of reliability was measured) Restriction in range Correlation between pretest and posttest Clustering design effect Additional notes or clarifications on study or coding variables
Statistical procedures and conventions
We will statistically synthesize effect sizes and assume a random effects model because of the potential for generalizability, using the inverse-variance method. We will therefore include the following statistical information using Review Manager 5.0 and/or the metafor package for R (Viechtbauer, 2010): a forest plot that has the effect size estimate its 95% confidence intervals for each study, the weighted mean effect size and its 95% confidence intervals, the value of Q, its df, and its related p value, the value of I2, the value of τ2, and funnel plots for outcomes with 10 or more studies. We intend to use a standardized mean difference effect size reported in Review Manager for continuous outcomes and odds ratios for dichotomous outcomes.
We plan to report outcomes separately by type of academic outcome, by the behavioral outcome, and by the students' age level if there are enough studies in each category. We will also analyze randomized control trials and quasi-experiment research studies separately if there are enough studies in each category to do so. Besides the study design moderator, we also intend to investigate whether the type of Montessori school (e.g., public or private) is a statistically significant moderator. We will conduct sensitivity analyses examining the effect of excluding the studies visually appearing to be outliers or clusters to help explain sources of heterogeneity.
Data extraction
Inclusion and exclusion criteria and data extraction for the coding sheet will be made independently by two reviewers for each article. When there are disagreements, the two reviewers will make a consensus decision. If there is no resolution, the final decision will be made by the first author of the protocol.
Treatment of qualitative research
We do not plan to include qualitative research.
Footnotes
Review Authors
| Name: | Justus J. Randolph, Ph.D. |
|---|---|
| Title: | Assistant Professor |
| Affiliation: | Mercer University |
| Address: | 3001 Mercer University Dr. |
| City, State, Province or County: | Atlanta, GA |
| Postal Code: | 30341 |
| Country: | U.S.A |
| Phone: | (678) 547-6519 |
| Mobile: | (404) 295-0755 |
| Email: |
|
|
|
|
|---|---|
| Title: | Postdoctoral Researcher |
| Affiliation: | Mercer University |
| Address: | 3001 Mercer University Dr. |
| City, State, Province or County: | Atlanta, GA |
| Postal Code: | 30341 |
| Country: | U.S.A. |
| Phone: | |
| Mobile: | |
| Email: |
|
|
|
|
|---|---|
| Title: | Postdoctoral Researcher |
| Affiliation: | Mercer University |
| Address: | 3001 Mercer University Dr. |
| City, State, Province or County: | Atlanta, GA |
| Postal Code: | 30341 |
| Country: | U.S.A. |
| Phone: | |
| Mobile: | |
| Email: |
|
|
|
|
|---|---|
| Title: | Public Services Librarian—Education Liaison |
| Affiliation: | Mercer University |
| Address: | 3001 Mercer University Dr. |
| City, State, Province or County: | Atlanta, GA, Dekalb County |
| Postal Code: | 30341 |
| Country: | U.S.A. |
| Phone: | (678) 547-6256 |
| Mobile: | (706) 255-1272 |
| Email: |
|
|
|
|
|---|---|
| Title: | Associate Professor, Department Chair |
| Affiliation: | Mercer University |
| Address: | 3001 Mercer University Dr. |
| City, State, Province or County: | Atlanta, GA |
| Postal Code: | 30341 |
| Country: | U.S.A. |
| Phone: | (678) 547-6376 |
| Mobile: | (404) 556-1675 |
| Email: |
|
|
|
|
|---|---|
| Title: | Director |
| Affiliation: | Athens Montessori School |
| Address: | 3145 Barnett Shoals Road |
| City, State, Province or County: | Athens, GA |
| Postal Code: | 30605 |
| Country: | U.S.A. |
| Phone: | (706) 548-8490 |
| Mobile: | – |
| Email: |
|
ROLES AND RESPONSIBILITIES
Systematic review methods: Justus Randolph
Statistical analysis: Justus Randolph
Data extraction: Anaya Bryson, Lakshmi Menon
Information retrieval: Stephen Michaels
Content: deb Rosenstein, Warren McPherson
Sources of Support
There is no financial or other support for the proposed review.
Declarations of interest
The first author's spouse is a Montessori teacher. One of the content experts is the Director at a private Montessori School.
Preliminary timeframe
The systematic review will be completed within two years from acceptance of the protocol. It is our hope that the data can be collected in the fall and spring semester of 2016-2017, the data analyzed by the summer semester of 2017, and the report written by the fall semester of 2017.
Plans for Updating the Review
Reviews should include in the protocol specifications for how the review, once completed, will be updated. The protocol specifications should include information on who will be responsible and the frequency with which updates can be expected. Justus J. Randolph will be responsible for updating the review every decade.