Abstract
Mixed methods approaches to research are gaining popularity in the social sciences. Although these approaches may be unfamiliar to many in our field, they can uniquely contribute to and enhance early childhood special education (ECSE) research. The purpose of this article is to orient ECSE researchers to the field of mixed methods social inquiry. We offer two examples of mixed methods. We define mixed methods and how mental models and paradigms influence these efforts, including a discussion of the distinctive purposes for applying mixed methods. Finally, we identify challenges to determining rigor and quality of mixed methods research and offer preliminary guidance to mitigate these challenges. Throughout, we encourage integrating rigorous mixed methods into ECSE scholarship.
To advance social science in early childhood special education (ECSE), researchers seek to clarify the scope of the experiences faced by young children, their families, and the professionals who support them. Researchers have developed methods to count and describe these experiences, including surveys (Jackson, 2009), interviews, observations, and unobtrusive measures (Brantlinger, Jimenez, Klingner, Pugach, & Richardson, 2005). Yet, often, the numbers derived from quantitative methods obscure the complex lived experience of the humans they count, whereas the narratives generated by qualitative methods offer insights into small samples but do not convey the scope of an issue (Odom et al., 2005). Similarly, intervention research is a cornerstone of education and ECSE research. Intervention research seeks to support children, families, and professionals in gaining new skills. Experimental research designs have been well established to identify the extent to which interventions influence behavior, observed learning, and other indicators of the intervention effect. Even so, the extent to which these interventions influence everyday practice lags, begging the question, “What are we missing?”
Mixed methods may provide a unique and innovative way to mitigate many of the challenges ECSE researchers face. However, in the top-ranked journals in ECSE (i.e., Topics in Early Childhood Special Education, Journal of Early Intervention, Infants and Young Children), mixed methods has rarely been published. Between the years 2007 and July 2019, only 8 out of 564 original research articles published in these journals purport to use mixed methods (i.e. 1.14%) (Corr, Snodgrass, Love, Scott & Kim, under review). The purpose of this article is to provide a brief overview of mixed methods, including its definition, the purposes for which researchers may choose to apply mixed methods, common challenges, and guidance for judging rigor and quality of mixed methods studies. We briefly introduce the key concepts of the mixed methods community in an attempt to orient ECSE researchers to this research method and to clarify common misconceptions.
The term “mixed methods” has been increasingly invoked in educational research to represent a broad and varied body of research endeavors. The field of mixed methods research has its own unique history, tradition, and preferred terminology and is itself still evolving (Creswell, 2011; Greene, 2007; Teddlie & Tashakkori, 2010). Notably, mixed methods conversations continue to display inconsistencies in definitions and applications of key concepts (Creswell, 2010). Mixed methods is commonly thought of as combining qualitative and quantitative methods within the same research study (Creswell, 2011; Teddlie & Tashakkori, 2010). However, such efforts do not represent the full range of possibilities in the mixed methods field. Researchers have advocated mixing not only methods, but also methodologies, theories, standpoints, and paradigm assumptions. Similar variation exists for other components of mixing methods as an educational and social research methodology.
We believe that mixed methods has great potential to contribute to the advancement of the science in ECSE. For example, mixed methods allows ECSE researchers to explore such questions as “How do ECSE and child welfare systems support young children with disabilities who also experience abuse and neglect?” (Corr, 2015) or “What practices are most effective in supporting families and educational teams to help young children learn to use augmentative and alternative communication?” (Snodgrass, 2016). As we will describe, mixed methods is both a way of thinking and a comprehensive, organized process for conducting research.
This article is intended to orient ECSE researchers to the major rationales for and components of mixed methods for research in ECSE. We believe that a mixed methods approach can meaningfully engage theory and practice, depth and breadth, contextuality and generality, and more. Our argument is importantly supported by examples drawn from our respective mixed methods research studies.
Considering Mental Models and Paradigms
Human beings approach life through a unique and evolving mental model, a set of “assumptions, understandings, predispositions, and values and beliefs” that guide their perceptions and decisions (Greene, 2007, p. 12). A person’s individual mental model influences what they consider to be true, credible, what they value, and the questions they ask. Educators and researchers experience a rich diversity of mental models. Teachers watch as parents are unimpressed by a beautifully graphed chart of their child’s progress on letter identification, even as their principal beams with pride at the demonstration of effective teaching. Early childhood educators negotiate Individualized Family Service Plan (IFSP) meetings where each team member has different priorities for and definitions of progress. Repeatedly, educators encounter the diverse ways in which people make meaning of their experiences.
In research, paradigms encompass assumptions about the nature of the social world, the character of the warranted knowledge constructed or generated about that world, and what forms of knowledge are most valued. Paradigms are a community’s shared ideas, beliefs, and values about the nature of reality and how knowledge is constructed (e.g., post positivist, constructivist, etc.). In this instance, sharing is considered a matter of degree as it is unlikely that all members of a community will share every component of a paradigm (Maxwell, 2009). In contrast, mental models are unique to individual researchers, and one person’s mental model is unlikely to directly align with any single paradigm. Although research methodologies are largely defined by paradigms, steeped in tradition and history, both paradigms and mental models influence the values that frame, guide, and thereby are advanced by our research, including the research questions researchers ask. These research questions then help us determine the methodologies applied.
The guidelines and methods used within each methodological tradition ascribe meaning and value to knowledge derived from a particular way of seeing the world. For example, single-subject/case research methodology ascribes value and meaning to, and thereby answers questions about, observable and measurable changes in behavior (Kazdin, 2011), whereas case study methodology ascribes value and meaning to, and thereby answers questions about, the contextual observations and interpretations of the researcher who examines the case (Stake, 1995).
Mixed methods research invites and values multiple mental models into research endeavors and makes space for exploring a different type of research question. As mixed methods evolves, researchers begin to think less in terms of the traditional dichotomies such as words versus numbers. Instead, researchers think more in terms of transformable units of information that happen to be initially generated in one of several possible forms (Teddlie & Tashakkori, 2009). By mixing methods, researchers purposefully choose to engage multiple mental models in their work and to value the knowledge generated by each, in service of better understanding the phenomenon being studied (Greene, 2007). The grand vision for mixed methods research defines a mixed methods way of thinking [as] a stance or an orientation toward social research and evaluation that is rooted in a multiplistic mental model and that actively invites to participate in dialogue—at the larger table of empirical inquiry—multiple ways of seeing and hearing, multiple ways of making sense of the social world, and multiple standpoints on what is important and to be valued and cherished. (Greene, 2007, p. 20)
Levels of Mixing
Within mixed methods studies, researchers choose to mix at different levels: methods, methodologies, or paradigms. Mixing at the methods level refers to research studies in which two or more different instruments or ways of collecting data (e. g., interviews and surveys) are employed during the research process. Mixing at the methodological level refers to research studies that use the logic of inquiry from two different traditions, for example, quasi-experimental research and case study research. Methodology in this context refers to the character of the research questions addressed, sampling logic, instruments used, criteria for data quality, preferred analyses, and reporting conventions. Mixing at the paradigmatic level is a complex and ongoing conversation among philosophers in the mixed methods community (see Greene, 2007). This complex conversation is outside the purview of this article.
The important differences between mixing at the level of method and mixing at the level of methodology can best be engaged through an example. Imagine two researchers—Helen and Marjorie—who have questions about the social development of several young children with developmental disabilities in Kindergarten. Both researchers have selected a case study methodology and accompanying qualitative methods of observation and interviewing to address their research question. Helen decides to add a semi-structured interview of the child’s parents, siblings, and nanny to enrich her observation data of her target children’s interactions and expand the focus of her research question. Marjorie decides to add a single-subject/case research methodology to her study, selecting one child for more intensive observational tracking as well as collecting additional behavioral rating data on this child from the teacher. Helen is mixing at the method level (case study and semi-structured interview); Marjorie is mixing at the methodological level (case study and single-subject/case research design).
Mixing at the Methods Level: Within Complementary Paradigms
Some paradigms complement one another quite easily, as the distinctions between them are small. For example, the paradigm that guides randomized controlled trials places value on statistically significant differences on validated measures of performance between groups of individuals who received an intervention and those who did not. Similarly, the paradigm that guides single-subject/case research designs places value on measures of observed behavior change within a single case who received an intervention that was systematically manipulated. Although different methods are used in randomized trials and single-subject/case research studies to gather evidence and draw conclusions about an intervention’s effect, both traditions adopt paradigms that privilege knowledge based on experimental manipulation of an independent variable, followed by observations of (hopefully) compelling changes in dependent variables. That is, both methods are derived from the same (or similar) methodological tradition.
Mixing at the Methodology Level: Across Divergent Paradigms
Other paradigms are not so easily reconciled (Greene, 2007). Consider, for example, claims of intervention effectiveness derived from a case study versus a randomized controlled trial. When mixing at the methodology and paradigm levels, the knowledge privileged by one methodology used is not necessarily credible to people who subscribe to the paradigm of the other methodology employed. Here, the methods used are derived from very different methodological traditions.
Purposes for Mixed Methods Research
Mixed methods can be useful in ECSE research (Creswell, 2011; Greene, 2007; Morse, 2010). A single method may be sufficient to address a researcher’s purpose. However, when considering mixed methods, researchers must decide why they are mixing. There are three basic purposes for mixing methods: (a) for development, (b) for conclusions, and (c) for expansion (Greene, 2007; Greene, Caracelli, & Graham, 1989). These purposes should be planned for in advance of a study but may change in response to study events. There are no formulas for mixed methods research; rather, researchers are tasked with making defensible methodological decisions and articulating those decisions to their audience.
In this section, we describe these purposes and offer examples from our own research studies. Corr (2015) used mixed methods to describe how young children with disabilities are supported in early intervention and child welfare systems. Snodgrass (2016) used mixed methods to examine the effects of an intervention to help a Kindergartener’s educational team support his augmentative and alternative communication (AAC) learning. In Table 1, we provide details on samples, research questions, methods, paradigms, and purposes for mixing from each study. We do not fully describe our studies; rather, we use them as points of reference to detail the purposes for which researchers may choose to apply mixed methods. Although other examples of mixed methods research in ECSE exist, we are not privy to the thinking and reasoning of these other researchers. We use examples from our own work because we can share insights into both the conceptualization and the mechanics of conducting it.
Examples of mixed methods to descriptive and intervention research.
Note. ECSE = early childhood special education; CW = child welfare; EI = early intervention; AAC = augmentative and alternative communication.
For Development
One purpose for mixed methods is to use the results of one method to develop the sampling plan, implementation procedures, and/or instrument for another method. When a mixed methods design is adopted for this purpose, the different methods are implemented sequentially and the mixing intentionally occurs across methods (Greene, 2007). For example, Corr (2015) mixed interview, survey, and case study methods for the purpose of development in a research study examining the provision of services for young children who have experienced abuse and neglect (see Table 1). To address her research questions, she intentionally examined the same phenomenon from the state, system, and local perspectives. To ensure that the data collected from participants across these levels were connected, she purposefully implemented each method sequentially, beginning with a survey of service providers (i.e., system-level participants). She then used the constructs from that survey to develop semi-structured interview questions used with state-level administrators. After the analysis, she used the interview and survey results to develop the data collection sources and key questions for the case study of a local family (e.g., weekly communication logs, interview questions). Thus, she decided to use the results of each method to inform the development of the instruments used in the subsequent method.
For Stronger Conclusions
Although mixed methods research always seeks stronger conclusions, some researchers may decide to employ mixed methods focused specifically on the conclusions they draw. Greene (2007) elucidated different purposes for mixing that focus directly on strengthening conclusions: (a) for triangulation or complementarity, (b) for initiation when divergence occurs, and (c) for planned initiation. We describe each of these purposes in detail next.
Triangulation or complementarity
When mixed methods is adopted for the purpose of triangulating conclusions, the researchers invoke different methods to examine the same concrete phenomenon, analyzing each method separately and mixing only to draw conclusions. Similarly, when mixed methods is adopted for the purpose of demonstrating complementary conclusions, the researchers invoke different methods to examine various facets of the same complex phenomenon. Unlike triangulation, researchers may choose to mix during the analysis (see Bazeley, 2018). Corr (2015) mixed methodologies for the purpose of complementarity. By mixing interview, survey, and case study methodologies across the state, system, and local perspectives, she described how professional role confusion impacts service delivery for young children with disabilities who have experienced abuse and neglect. Although professional role confusion appeared within the results of each method, the use of a mixed methodology underscored the unique but complementary differences around role confusion manifest at the different levels of the system established to support children who had experienced abuse and neglect.
Initiation when divergence occurs
Researchers who use mixed methods for the purposes of strengthening conclusions may hope that their results across methods will triangulate or complement one another. However, this will not always be true. Whenever researchers use mixed methods for the purpose of drawing stronger conclusions, they may find themselves with divergent or contradictory findings across methods. When this occurs, the purpose of mixed methods shifts from triangulation or complementarity to initiation (Greene, 2007; Maxwell & Loomis, 2003). The premise here is that divergent findings, rather than being viewed as problematic, may provide the researchers with an opportunity to initiate new understanding about the phenomenon they are studying. Why do the findings diverge? What can be understood about what is being studied from these contrary results?
For example, Snodgrass (2016) mixed single-subject/case research (Kazdin, 2011) and case study (Stake, 1995; Yin, 2014) methodologies to examine an intervention designed to support an educational team in teaching a child to communicate via an AAC system (see Table 1). Her single-subject/case research data indicated that the child had made few gains in using AAC. However, in interview and self-report data, the participants reported feeling that the child had made progress and benefited from the intervention and provided anecdotes about changes in the child’s communication. By mixing single-subject/case research and case study data sources during the analysis, these divergent findings emerged and gave the research team the opportunity to initiate new understanding about what constitutes growth in AAC and how the intervention impacted the child’s learning. They would have likely missed such understandings if they had only employed one method to examine this phenomenon.
Planned initiation
Researchers may also invoke mixed methods to intentionally reveal dissonance and divergence for the purpose of initiation (Greene, 2007). For example, Corr (2015) asked participants in her mixed methodology study about how research impacts their work supporting young children with disabilities who have experienced abuse and neglect. She hypothesized that perspectives would differ between system-level administrators and local practitioners. By mixing methodologies, Corr planned for initiation by including this topic in her data collection tools across methodologies. When the data from participants verified this hypothesis, she then had the opportunity to initiate new understandings about the role of research in policy and practice for supporting families and children who have experienced abuse and neglect.
For Expansion
Mixed methods can also be used to expand the focus of a study to include the examination of multiple phenomena. When researchers use mixed methods for expansion, they mix to expand their focus to include two or more phenomena (Greene, 2007). For example, Snodgrass (2016) was interested in the factors that impact the communication skills that young children who are learning to use AAC develop. She hypothesized that two distinct phenomena may be at play: (a) how the educational team function as they work together to support the child’s learning (Phenomenon 1) and (b) the team’s instructional effectiveness when working directly with the child (Phenomenon 2). By conducting both a single-subject/case research experiment and a case study, Snodgrass could explore how these two phenomena interacted to influence the AAC learning of the child who participated. This mixing strengthened conclusions about how the intervention she provided to the team affected both their functioning as a team (Phenomenon 1) and their instructional competence (Phenomenon 2), and how these, in turn, influenced the child’s AAC learning. A single method would not have allowed exploration of this complex interaction.
Challenges and Guidance for Determining Rigor and Quality
Mixed methods research allows for innovative design, variation within design, and diverse composition of populations studied in research. This is an extraordinary benefit for ECSE researchers who work with diverse and complex populations and social issues. Furthermore, the field of mixed methods leaves much room for innovation and exploration into how mixed methods can be wielded to address the pressing and often complex issues ECSE researchers tackle (e.g., inclusion, recommended practices, systems changes). As Lieber and Weisner (2010) stated, All methods are intended to (a) bring us closer to the phenomenon of interest, (b) help us discover truths about the world, (c) produce research findings that are meaningful and valuable to the social sciences, (d) provide findings that are “believable” and supportive of the research claims, and (e) instill confidence in ourselves as researchers in the audiences to whom we disseminate our work. (p. 261)
With such an endeavor, however, come unique challenges and opportunities in judging rigor and quality. What constitutes a strong mixed methods study? Are the procedures sufficiently rigorous? Are the findings valid and conclusions warranted?
To answer these questions, some may want to develop a straightforward quality checklist to guide peer reviewers and consumers of mixed methods in making judgments of rigor and quality (Tashakkori & Creswell, 2007). However, given that there are innumerable levels, purposes, and designs for mixed methods studies and that all research requires careful, individualized evaluation to rule out threats to validity, checklists cannot be used to adequately assess the rigor and quality of mixed methods studies (Sandelowski, 2003). Instead, mixed methodologists have encouraged a critique of each mixed methods research study on its own merits (O’Cathain, 2010; Sandelowski, 2003). The following considerations can guide researchers, reviewers, and consumers in applying individualized and dynamic critiques that respond to the mixed methods design at hand. For additional information and resources on mixed methods research designs, see Table 2. We outline four critical considerations that have been established and discussed in the mixed methods field. We offer these not as a comprehensive list, but, instead, as a set of focused considerations for rigorous and high-quality mixed methods research.
Additional Mixed Methods Resources.
First, the rigor and quality of mixed methods studies are influenced by the mental models of the researchers conducting a study (Bryman, 2006). Even if researchers do not explicitly disclose their mental models, they should better attend to their influences on their work, alert reviewers and consumers to these influences, and encourage representation of a wide variety of philosophies within disseminated rigorous research (O’Cathain, 2010). Therefore, researchers should both identify and articulate the mental models that inform their design, analysis, and interpretation so reviewers and consumers can take this into consideration.
The second step toward rigor and quality for a mixed methods study is determining the fit of a mixed methods approach to the research questions posed. When mixed methods are employed well, they can (a) bring us closer to a full representation of social phenomena (Lieber & Weisner, 2010); (b) more richly, authentically, and appropriately represent the true complexity of behavior as they occur in natural social contexts (Johnson & Onwuegbuzie, 2004; Lieber, 2009; Lieber & Weisner, 2010); and (c) enhance our ability to discover, understand, and communicate findings to a wide range of audiences (Lieber & Weisner, 2010). In spite of these rich opportunities, not all research questions benefit from the application of a mixed methods study. Instead, researchers must carefully identify the purpose for selecting the methodology that allows them to best answer their research questions. ECSE researchers should articulate and justify why a mixed methods design is best suited to answer their research questions (Johnson & Onwuegbuzie, 2004).
Third, in mixed methods research, rather than depending on existing checklists of quality criteria, researchers should consider domains of quality (Bryman, 2006; O’Cathain, 2010). These include (a) planning quality, (b) design quality, (c) data quality, (d) interpretive rigor, (e) inference transferability, and (f) utility (O’Cathain, 2010). Each domain should be considered during each stage of the research process, including planning, undertaking, interpreting, disseminating, and application in the real world (O’Cathain, 2010). ECSE researchers should plan for and report on efforts across these domains and time points when applying mixed methods to their research, in addition to the guidance specific to each method included in a mixed methods study from our field (e.g., Brantlinger et al., 2005; Gersten et al., 2005; Horner et al., 2005).
Finally, the language used in mixed methods research can be challenging. Well-established terms and language exist for discussing the quality of a particular method in special education research (Brantlinger et al., 2005; Gersten et al., 2005; Horner et al., 2005; Teddlie & Tashakkori, 2003; Thompson, Diamond, McWilliam, Snyder, & Snyder, 2005), but not for a mix of methods/methodologies. Terms such as validity, trustworthiness, and credibility have specific meanings within respective methodologies. In response, the ECSE research community should use clear descriptions, rather than refer to easily misinterpreted terms, to facilitate the efforts of peer reviewers and consumers in determining quality (Teddlie & Tashakkori, 2003). This includes clearly describing the purposes for mixing, the tools used to conduct that mixing, and the specific steps taken to rigorously evaluate the produced results. These descriptions can draw concepts and arguments from multiple research traditions, as appropriate, rather than assuming a shared understanding of terminology.
Conclusion
Mixed methods has been thrust into the consciousness of researchers across disciplines, including special education (Klingner & Boardman, 2011; Teddlie & Tashakkori, 2009). Although much of what it means to use mixed methods in research continues to evolve, common ideas bind this community together to construct a distinct methodological identity (Tashakkori & Teddlie, 2010). This presents exciting opportunities for discovery in ECSE research. Mixed methods affords early childhood researchers unique opportunities to examine complex issues and can support many different goals in research (Greene, 2007). When applied purposefully and rigorously, mixed methods research can stand equally with single-method inquiry in contributing to our understanding and evidence base about the mosaic of early childhood interests. As a field, the challenges of addressing rigor and quality of mixed methods designs also yield great opportunity to wield them to strengthen and expand our understanding of supporting young children, families, and early childhood professionals (O’Cathain, 2010). We offer this introduction in the hopes that it will propel mixed methods research in the methodological conversation in our field. By defining common terms, detailing the purposes for which researchers may choose to apply mixed methods research, and offering an introductory exposition of common challenges and guidance when determining rigor and quality, we look forward to ongoing, robust conversation, demonstration, and growth of the rich methodological traditions that guide and shape the ECSE field.
Footnotes
Acknowledgements
We are grateful for the careful and constructive feedback from the JEI reviewers and associate editor. This manuscript is stronger because of critical feedback.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
