Operationalizing and Expanding the Rigorous Mixed Methods Framework for Systematic Reviews: The Case of Inclusive Education

Abstract

Rigor evaluation in mixed methods research is a growing need. Linking rigor to Hong and Pluye’s (2019) concepts of methodological and reporting quality, the purpose of this article is to operationalize and expand Harrison et al.’s (2020) Rigorous Mixed Methods Framework. Drawing from a systematic methodological review of 66 inclusive education studies, we (a) operationalize each element of the original framework, and (b) present arguments for two additional advanced elements, paradigm and ethics. Contributions to the field of mixed methods include situating the concept of rigor in the quality literature, expanding a rigor tool by developing guiding questions, and offering both conceptual and empirical justification for including paradigm and ethics in the rigor tool.

Keywords

inclusive education mixed methods rigor expanded rigorous mixed methods framework systematic methodological review

Introduction

The use of mixed methods approaches in many fields is growing (Molina-Azorin & Fetters, 2016), and its rigorous application is an increasing concern (Fàbregues & Molina-Azorín, 2017; Howell Smith & Shanahan Bazis, 2021). Guetterman et al. (2023) defines mixed methods quality as “how well a mixed methods study was conducted through scientifically accepted design and procedures” (p. 7). For conducting systematic methodological reviews (i.e., the methodological review of mixed methods research [Howell Smith & Shanahan Bazis, 2021]), evaluation of conceptual, methodological, and reporting quality is recommended (Hong & Pluye, 2019; Pluye & Hong, 2014). Quality therefore encompasses conceptualization of the research phenomenon to generate new insights, the conduct of the study using scientifically accepted procedures, and reporting of the study. The literature on mixed methods research quality identifies core criteria, contextualizes criteria for diverse disciplines and designs, and constructs criteria for systematic reviews (Guetterman et al., 2023). Existing criteria focus on mixed methods research reporting (e.g., Fàbregues & Molina-Azorín, 2017; Hong & Pluye, 2019; Leech, 2012; Leech & Onwuegbuzie, 2010; Levitt et al., 2018; O’Cathain, 2009; Pluye & Hong, 2014) and rigor (Harrison et al., 2020). Choosing an appropriate rigor tool for a systematic methodological review is compounded by ambiguity regarding the distinction between rigor and quality and the amount of guidance for applying criteria (Fàbregues & Molina-Azorín, 2017; Guetterman et al., 2023; Howell Smith & Shanahan Bazis, 2021). Harrison et al. (2020) identified rigor criteria as a subset of quality criteria; defining rigor as the specific steps taken to conduct and report on the study (Wisdom et al., 2012), they presented the Rigorous Mixed Methods Framework.

In conducting a systematic methodological review in our field of inclusive education, we chose to apply the Rigorous Mixed Methods Framework for three reasons. First, Harrison et al.’s (2020) framework is the only tool that has explicitly identified criteria for evaluating rigor. Second, its criteria align with common accepted criteria (Fàbregues & Molina-Azorín, 2017; Hirose & Creswell, 2023); five of the framework’s six criteria align with Hirose and Creswell’s (2023) core quality criteria and the last was identified as a possible limitation to their list. Third, the framework’s continuum allows for a more nuanced application of criteria rather than a dichotomous response; detailed insights into the study’s strengths and weaknesses are possible. Moreover, since Harrison et al. (2020) constructed each criterion along the continuum, the framework provides some guidance on how to evaluate low, medium, and high levels of rigor.

In applying the framework to our own work, we discovered three challenges. First, Harrison et al. (2020) conceptualized rigor for each criterion as a continuum with three categories: high, medium, and low and then used these evaluations to classify studies on a five-point continuum (high, medium-high, medium, low-medium, and low). Characteristics of these five categories were not explicitly defined which could lead to high variability when reporting rigor. Second, due to our different methodological orientations, we applied the criteria differently, affecting interrater agreement. Given that we both have mixed methods training and experience designing and publishing mixed methods studies, it is possible that researchers less familiar with mixed methods could encounter additional challenges (Fabregues et al., 2019). Third, given the importance of paradigms and ethics in the mixed methods literature (Cain et al., 2019; Creswell & Plano Clark, 2007; Hesse-Biber, 2010; Hirose & Creswell, 2023; Poth, 2018; Teddlie & Tashakkori, 2009), we determined it was important to include these criteria in an expanded rigor framework. Since most quality literature focuses on conceptual arguments (Hirose & Creswell, 2023), we sought to provide empirical evidence for the addition of these criteria.

Building on the work of Harrison et al. (2020), the methodological aim of this article is to operationalize Harrison et al.’s (2020) framework and to justify the addition of two advanced elements. The expanded framework includes explicit criteria for conducting systematic methodological reviews. We begin with a definition of mixed methods and a discussion of its application within the field of education. Subsequently, we describe and operationalize the existing Rigorous Mixed Methods Framework (Harrison et al., 2020), providing examples from the inclusive education field. We then present the conceptual and empirical evidence for two additional elements. Lastly, we discuss the strengths and limitations of the Expanded Rigorous Mixed Methods Framework and considerations for future work.

Definition of Mixed Methods

We use the definition of mixed methods underlying the Rigorous Mixed Methods Framework. Specifically, mixed methods research combines elements of quantitative and qualitative approaches to gain greater understanding of the studied phenomenon, with the expectation that individual approaches are applied with the same rigor as when applied singly. (Harrison et al., 2020). Of note, our methodological review is also consistent with current definitions of mixed methods research that additionally emphasize the importance of integration and the development of meta-inferences across qualitative and quantitative findings (Fàbregues & Molina-Azorín, 2017; Guetterman et al., 2015).

Mixed Methods in Inclusive Education

Internationally, inclusive education is defined as effectively educating all children, including those with disabilities, in general education classrooms together with their peers; although students with specific needs may receive specialized services, such services are integrated into the general curriculum (Ainscow et al., 2006). Given that equal and quality education is a global priority requiring an understanding of statistics and lived perspectives (International Institute for Educational Planning, 2020), mixed methods are relevant. Methodological reviews in the related field of special education (i.e., focused on the education of children with disabilities, occurring in settings ranging from general education to specialized classrooms or schools [Ainscow et al., 2006]) suggest that researchers are not fully capitalizing on the potential of mixed methods approaches. Specifically, such reviews reveal that less than 1% of articles explicitly identify as mixed methods and insufficient reporting hindered evaluation of their quality (Corr et al., 2020; Onwuegbuzie & Corrigan, 2018). Because the fields of special and inclusive education, although related, are distinct, we undertook a systematic methodological review in inclusive education (Kutscher & Parey, 2022). Findings from our review indicated overall medium rigor among the documents and suggested that researchers were not maximizing the potential of mixed methods (Kutscher & Parey, 2022). For more details, see Kutscher & Parey (2022).

Rigor in the Context of Quality

Conceptualizing mixed methods research quality is a prerequisite for identifying appropriate criteria (Fabregues et al., 2019), yet despite growing literature on mixed methods research quality, there is no consensus on its conceptualization (Guetterman et al., 2023). Although some argue that developing common criteria is difficult due to differences across disciplines, agreement on quality criteria is essential to avoid poor research practices, such as ineffective integration of research strands (Fabregues et al., 2019; Guetterman et al., 2023). Many tools focus on reporting quality (Guetterman et al., 2023); however, Hong and Pluye (2019) conceptualized quality in three dimensions: conceptual, methodological and reporting. Our systematic methodological review focused on rigor, and Harrison et al.’s (2020) definition of rigor (i.e., the specific steps taken to conduct and report on the study [Wisdom et al., 2012]) is aligned with Hong and Pluye’s (2019) concepts of methodological and reporting quality. Specifically, methodological quality pertains to how a study is conducted (i.e., methodology, methods, and validity issues), and reporting quality refers to transparency, accuracy, and completeness in reporting (Hong & Pluye, 2019). The alignment of rigor with methodological and reporting quality supports Harrison et al.’s (2020) claim of rigor criteria as a subset of quality criteria. Harrison et al.’s (2020) framework considers six elements: four primary (data collection, data analysis, data integration, mixed methods design type) and two advanced (aims and purpose, writing).

Methodology

Drawing on a systematic methodological review of 66 inclusive education documents (Figure 1), we illustrate the operationalization and expansion of Harrison et al.’s (2020) Rigorous Mixed Methods Framework.

Figure 1.

PRISMA Diagram illustrating the screening process for the systematic methodological review. Source: Modified from Kutscher & Parey (2022, p. 4).

Development Process for the Expanded Framework

Operationalization of Original Elements

We developed a Google form to support data extraction; items aligned with Harrison et al.’s (2020) six elements and the guidelines for applying each along the continuum. In applying the framework, we observed that some criteria were open to interpretation, which resulted in inconsistent rigor ratings. For example, for data collection, while Harrison et al. (2020) noted the importance of “reporting…specific data collection procedures for both qualitative and quantitative data strands” (p. 6), they provided no additional guidance. Hence, our application of the original criteria presented some challenges, especially since we had differing methodological orientations (e.g., one author determined that a report of data collection procedures was sufficient—“high rigor”—and the other determined limited reporting—“medium rigor”). Given the amount of discussion required to resolve discrepancies while reviewing a subset of articles to reach consensus, we determined that further work to clarify the criteria was necessary before proceeding with the review. Therefore, we operationalized the original criteria as described in this article.

We identified 5–10 documents to review weekly (i.e., independently applied our own interpretations of each rigor rating), focusing on one specific element of the framework at a time (Harrison et al., 2020). During meetings, we used our agreements and disagreements to determine characteristics of each rigor rating, developing the 23 questions in Table 1, Table 2, Table 3, Table 4, Table 5, and Table 6. If criteria within the original framework required additional clarification, we referred to seminal mixed methods literature (e.g., Greene et al., 1989; Guetterman et al., 2015; italicized items in Table 1, Table 2, Table 3, Table 4, Table 5, Table 6 represent additions to the original framework). Moreover, guided by the concepts of methodological and reporting quality, we developed criteria along a 5-point continuum to move beyond transparency by including characteristics that demonstrate evidence of rigorous application of the research process. We applied an iterative process to ensure consistency; as new insights were gained, we re-examined all previously reviewed documents and recorded any changes. We then re-reviewed all 66 documents to consistently apply the operationalized framework. Interrater agreement ranged from 78.8% for the data integration element to 100% for the writing element; agreement for data collection, aims and purpose, data analysis, and mixed methods design were 87.9%, 89.4%, 95.5%, and 98.5%, respectively. Disagreements regarding data integration occurred when linking was not stated or inconsistent with the mixed methods design type. Resultantly, we clarified items a and b (Table 4), reevaluated all documents, and discussed the items to reach consensus. To determine document rigor, we transformed the element ratings into a scale of 1 (low) to 5 (high) and added the ratings across the six elements to calculate the original overall rigor score. Using the original overall rigor score, we classified each document into low (original score of 6), low-medium (7–14), medium (15–21), medium-high (22–29), and high rigor (30) categories (Kutscher & Parey, 2022).

Table 1.

Criteria for Aims and Purpose.

	High	Medium-High	Medium	Low-Medium	Low
a. Is a description of the purpose/rationale included?	Yes	Yes	Yes	No	No
b. Was the purpose/rationale stated?	Yes	Yes or can be inferred	Yes or can be inferred	No	No
c. Was a mixed methods research question included?	Yes	A Yes B No	No	A Yes B No	No
d. Was a discussion of the value of using mixed methods based on the findings included?	Yes	A No B Yes	No	A No B Yes	No

Note. Italics represent item added beyond Harrison et al.’s (2020) original framework. For medium-high rigor, the document must meet item a, b, or c OR meet item a, b, or d. For low-medium rigor, the document must meet either criteria c OR d.

Table 2.

Data Collection.

	High	Medium-High	Medium	Low-Medium	Low
a. How is the sampling procedure described?	Both strands: Strong	Strand 1: Strong Strand 2: Strong description in 2 criteria, limited description in 3rd	Both strands: Strong description in 2 criteria, limited description in 3rd	Both strands: Limited or no description of two criteria; strong description in 3rd OR both strands: Limited or no description of two criteria; limited description in 3rd	Both strands: No description
b. How is (are) the instrument(s) described?	Both strands: Strong	Strand 1: Strong Strand 2: Strong description in 2 criteria, limited description in 3rd	Both strands: Strong description in 2 criteria, limited description in 3rd	Both strands: Limited or no description of two criteria; strong description in 3rd OR both strands: Limited or no description of two criteria; limited description in 3rd	Both strands: No description
c How is (are) the type(s) of data described?	Both strands: Strong	Strand 1: Strong Strand 2: Strong description in 2 criteria, limited description in 3rd	Both strands: Strong description in 2 criteria, limited description in 3rd	Both strands: Limited or no description of two criteria; strong description in 3rd OR both strands: Limited or no description of two criteria; limited description in 3rd	Both strands: No description

Note. During data charting, there were six questions for this element as each question was asked separately for the qualitative and quantitative strands.

Table 3.

Data Analysis.

	High	Medium-High	Medium	Low-Medium	Low
a. How is data analysis described in Strand 1?	Strong	A. Limited B. Strong C. Limited	Limited	Limited	No
b. How is data analysis described in Strand 2?	Strong	A. Limited B. Limited C. Strong	Limited	No	No
c. How did mixed methods support the overall analysis?	Strong	A. Strong B. Limited C. Limited	No	No	No

Note. Italics represent item added beyond Harrison et al.’s (2020) original framework. For medium-high rigor, the document must meet (A) limited data analysis in strands 1 and 2, strong mixed analysis; (B) strong data analysis in strand 1, limited strand 2 and mixed analyses; OR (C) strong data analysis in strand 2, limited strand 1 and mixed analyses.

Table 4.

Data Integration.

	High	Medium-High	Medium	Low-Medium	Low
a. Are data strands linked specific to the design (e.g., do strands merge, explain, or build depending on design)?	Yes	Yes	Yes	A Yes B No	No
b. Are points of linking stated?	Yes	Yes	Yes or No	No	No
c. Are joint displays or data comparisons used?	Yes	Yes	Yes	A No B Yes	No
d. Are points of linking described?	Yes	No	No	No	No

Note. Italics represent item added beyond Harrison et al.’s (2020) original framework. For low-medium rigor, the document must meet (A) design-specific linking but no joint display OR (B) joint display but no design-specific linking.

Table 5.

Mixed Methods Design Type.

	High	Medium-High	Medium	Low-Medium	Low
a. Is a mixed method design identified?	Yes	Yes	No—both qual and quant strands are discussed	No—both qual and quant strands are discussed	No—had two strands but only one strand is discussed
b. Is a diagram used to display the design type?	Yes	No	No	No	No
c. Is the design type stated?	Yes	Yes	Can be inferred	No	No

Note. Italics represent item added beyond Harrison et al.’s (2020) original framework.

Table 6.

Elements of Writing.

	High	Medium-High	Medium	Low-Medium	Low
a. Was the study identified as mixed methods in the title, abstract, keywords, and/or paper?	Yes	Yes	Yes*	Yes	No
b. Was mixed methods literature cited?	Yes	Yes	Yes*	No	No
c. How was mixed methods literature cited?	Discussion of mixed methods literature	2+ citations	1 citation*	0 citations	0 citations

Note. Italics represent item added beyond Harrison et al.’s (2020) original framework. * = divergence from Harrison et al.’s (2020) criteria.

Additional Elements

We justify the inclusion of two additional advanced elements, paradigm and ethics, in the expanded framework. First, we provide conceptual arguments for their inclusion. Second, drawing from seminal literature (Cain et al., 2019; Creswell & Plano Clark, 2007; Hall, 2013; Hesse-Biber, 2010; Poth, 2018; Shannon-Baker, 2016; Stadnick et al., 2021; Teddlie & Tashakkori, 2009), we developed criteria for the additional elements. As with the original elements, we independently reviewed all 66 documents and met to compare responses and resolve discrepancies. Interrater agreement for paradigm and ethics was 92.4% and 97.0%, respectively. The expanded overall rigor score was computed based on eight elements (i.e., the original six elements plus paradigm and ethics). The expanded overall score ranged from 8 to 40 with 8 and 40 representing overall low and overall high rigor, respectively. As in Kutscher & Parey (2022), the expanded overall score was divided into approximate thirds to represent low-medium (9–19), medium (20–29), and medium-high (30–39) rigor.

Finally, to provide empirical evidence beyond conceptual arguments for the importance of paradigm and ethics in evaluating rigor, we employed regression analysis using the original overall rigor score as the dependent variable and the paradigm and ethics rigor scores as independent variables. Since the original overall rigor score was synonymous with count data with an upper bound, a negative binomial quasi maximum likelihood estimator was employed (Wooldridge, 2002). For the regression analysis, the individual ratings were coded so the lowest count was zero (i.e., the original overall rigor score now ranged from 0 to 24). As is standard for non-linear regression models, the marginal effects, calculated at the means of the independent variables using the delta method (Wooldridge, 2002) are reported.

Findings

Operationalization of Original Elements

The following is a description of the data extraction form we created to support data charting, presented by rigor element. An example of high rigor is also briefly presented based on documents rated with an original overall rigor score of high or medium-high in our review (Kutscher & Parey, 2022).

Aims and Purpose

The original Harrison et al. (2020) framework specified that “presenting the aims and purposes involves providing a clear rationale for conducting a mixed methods study, including a mixed methods research question, and discussing the value of mixed methods research” (p. 6). From this we extracted three questions (Table 1, items a-c). Given the literature of various typologies (e.g., Greene et al., 1989), we added a fourth question (Table 1, item d). The responses to these four questions were used to evaluate rigor for aims and purpose.

An example of high aims and purpose rigor is demonstrated in Coombes et al.’s (2016) article on implementing the Good Behaviour Game (GBG). The researchers explicitly stated the purpose (i.e., triangulation), included a mixed methods research question, and regarding value, noted that while findings were consistent with the literature, “interview results also highlighted the substantial practical challenges associated with implementing and using the GBG” (p. 1).

Data Collection

Harrison et al.’s (2020) framework defined high rigor in data collection as “[including] the reporting of specific data collection procedures for both qualitative and quantitative data strands (e.g., sampling procedures, types of data to be collected, and instruments used in data collection)” (p. 6). We extracted three questions related to Harrison et al.’s element, to be answered independently for quantitative and qualitative strands (Table 2).

To support Harrison et al.’s (2020) call for the “specific” reporting of data collection procedures as well as methodological quality, we further operationalized the definition to describe strong reporting of quantitative and qualitative data collection procedures. A strong description of quantitative sampling procedures, included information about sample size, relationship between sample and population size (e.g., margin of error), and sampling technique. A strong description of qualitative sampling procedures provided information on sample size, achievement of saturation, and sampling technique (Creswell, 2014; Sale & Brazil, 2004). A strong description of the quantitative instruments included information on the items/questions in the instrument and information on the instrument’s reliability, while information on the items/questions and type of protocol (e.g., semi-structured) were identified for strong descriptions of qualitative instruments (Creswell, 2014; Sale & Brazil, 2004). Finally, strong descriptions of data types referred to explicit reporting of how quantitative (e.g., survey) and qualitative (e.g., focus groups, interviews) data were collected. Studies that mentioned sampling, instruments, and/or data type but did not fully describe the item were classified as “limited,” and studies with no information were categorized as “none.” Assessments of strong, limited and none for each question were then used to evaluate data collection rigor (Table 2).

An example of high data collection rigor is Wood’s (2017) dissertation on secondary school teachers’ attitudes and self-efficacy towards inclusive education. The researcher described the data collection instruments and included copies in the appendices. For the quantitative sampling procedure, detailed descriptions on the possible number of participants and the selection criteria were provided, while for the qualitative strand, steps to obtain a nested, representative sample of respondents were outlined.

Data Analysis

The original Harrison et al. (2020) framework defined high data analysis rigor as “the reporting of analysis procedures for both qualitative and quantitative data strands” (p. 6). From this, we extracted two questions (Table 3) giving consideration to three items each for both strands. For the quantitative strand, considerations included the use of (i) descriptive and/or inferential statistics, (ii) tools, and (iii) types of statistical tests (Creswell, 2014; Sale & Brazil, 2004). For the qualitative strand, considerations included information on the (i) codes and themes, (ii) tools, and (iii) procedure (Creswell, 2014; Levitt et al., 2018; Sale & Brazil, 2004). For both strands, the overall description was evaluated as “strong” (if all three items were reported), “limited” (if at least one or two items were reported) or “none” (if no items were reported).

In Harrison et al.’s (2020) framework, medium rigor differed from high rigor if “it is unclear how mixed methods are used to support the overall analysis” (p. 6), therefore, we added a third question to operationalize this distinction: How did mixed methods support the overall analysis? For this question, we evaluated whether the researchers reported a plan to use data transformation, compare quantitative and qualitative results, or for sequential designs, link the qualitative and quantitative strands (Creswell & Plano Clark, 2018). The answer to the question was also evaluated as “strong” (the plan was explicit, detailed, and consistent with the design type), “limited” (there was mention of mixed analysis but no description or the description was inconsistent with the design type) or “none” (no information). Altogether, answers to the three questions formed the basis for evaluating data analysis rigor (Table 3).

An example of high data analysis rigor is reported in Parey (2020), an article using a multiphase design to investigate accommodations for including children with disabilities at schools. For the initial qualitative strand, the description included a referenced analysis technique and use of qualitative findings to develop the quantitative instrument. For the quantitative strand, the researcher addressed the issue of missing observations before describing the analytical procedure and resulting statistics. In Parey’s (2020) second qualitative strand, which occurred concurrent to the quantitative strand, the author described how qualitative data were transformed into and analyzed as quantitative data.

Data Integration

Harrison et al. (2020) defined high rigor in data integration as “the linking of both data strands. Depending on the design type, both data strands are either merged or one data strand is used to explain, or build from, the other. Joint displays and/or data comparisons are utilized” (p. 6). We identified two questions corresponding to Harrison et al.’s definition (Table 4). Because the inclusion of clear and detailed information constitutes reporting quality, we added two additional questions relating to the explicit linking of the strands and the explicit reporting of the link (e.g., Creswell & Plano Clark, 2018). Data integration rigor was evaluated from the responses to these four questions (Table 4).

Engelbrecht and Savolainen (2018), an article on teachers’ attitudes, illustrate high data integration rigor. The researchers provided a visual representation of the research process and consistent with the study’s explanatory sequential design, linked the strands by “purposefully selecting participants” and “developing interview questions based on quantitative findings” (Engelbrecht & Savolainen, 2018, p. 669). They also provided a joint display showing the interaction between the quantitative and qualitative findings.

Mixed Methods Design Type

The original Harrison et al. (2020) framework specified high rigor in mixed methods design type as including “a mixed methods design type (e.g., sequential explanatory)” and using “a diagram to show the design type” (p. 6), from which we extracted two questions (Table 5). To improve reporting quality, we added a third question related to the explicit identification of the design type. These three questions were used to evaluate mixed methods design type rigor (Table 5).

Desutter’s (2015) dissertation on teachers’ perceptions of preparedness and competence is a good example of high mixed methods design type rigor. Using an explanatory sequential design, the researcher provided a design diagram, which included information on the procedure, products, and corresponding timelines.

Elements of Writing

The original Harrison et al. (2020) framework identified two characteristics of high writing rigor, namely, the inclusion of mixed methods literature references and the identification of the study as mixed methods in the title, abstract, or paper (p. 6; two questions, Table 6). Importantly, our definition of medium rigor diverged from Harrison et al. Specifically, Harrison et al. determined that a study reflects medium rigor if it “[includes] a discussion of mixed methods, but fails to cite any mixed methods literature, [and] does not identify the study as mixed methods” (p. 6). Our definition of medium rigor required that studies explicitly identify as mixed methods and include at least one reference to mixed methods literature, defined as a reference (e.g., journal article, textbook) with a title indicating that it was focused on mixed methods research approaches. Documents reflecting high writing rigor—in addition to identifying as mixed methods and citing mixed methods literature—included a discussion of mixed methods literature (i.e., beyond inserting one citation; Table 6.)

Walker (2017) used the term “mixed methods” in the title, abstract, and throughout their dissertation and included a discussion of the mixed methods literature with at least 10 mixed methods references (e.g., when describing the rationale, research design, or discussing integration).

Additional Advanced Elements: Paradigm and Ethics

This section addresses the two additional elements: paradigm and ethics. The conceptual arguments and data extraction questions for each are presented first, followed by the empirical results.

Conceptual Arguments

Top researchers in the field of mixed methods define paradigm as a worldview (Teddlie & Tashakkori, 2009; Creswell & Plano Clark, 2007) which are stances adopted on ontology, epistemology, axiology and methodology (Hall, 2013). Conscious use of paradigm guides the decisions of researchers throughout the research process (Levitt et al., 2018; Shannon-Baker, 2016) and its description is an important component in reporting quality (Fàbregues & Molina-Azorín, 2017; Howell Smith & Shanahan Bazis, 2021). Importantly, Hall (2013) identifies three possible paradigm stances for mixed methods research: an a-paradigmatic, multiple paradigm stance, and single paradigm stance. While arguing against an a-paradigmatic stance, Hall (2013) advocates for using multiple paradigms (where researchers draw on more than one paradigm) or a single paradigm stance, such as pragmatism, transformative-emancipatory paradigm, or critical realism (Hall, 2013; Shannon-Baker, 2016). From these considerations, we identified two questions for evaluating rigor in paradigmatic stance: (a) Is a paradigm stated? and (b) How is the paradigm described? Consideration was given to whether a paradigm was identified and its components (ontology, epistemology, axiology and methodology) were discussed for no, one, or both strands (Table 7).

Table 7.

Paradigms.

	High	Medium-High	Medium	Low-Medium	Low
a. Is a paradigm stated?	Yes	Yes	A. Yes B. For one strand	For one strand only	No
b. How is the paradigm described?	Description of ontology, epistemology, axiology and methodology and link between paradigm and methods	Description of some items: Ontology, epistemology, axiology and methodology	A. No description B. Description for one strand	No description	No description

Note. For medium rigor, the document must meet (A) paradigm stated but not described OR (B) paradigm stated and described for one strand.

Research ethics has been a major consideration since the late 1970s when the Belmont Report was issued. This also applies to mixed methods research (Hesse-Biber, 2010; Poth, 2018; Stadnick et al., 2021). Since ethical considerations should inform the steps taken by researchers and are important for reporting quality (Fàbregues & Molina-Azorín, 2017; Howell Smith & Shanahan Bazis, 2021), a demonstration of ethical inquiry is needed in high-rigor methodological practice (Cain et al., 2019). Cain et al. (2019) identified four themes related to how mixed methods researchers discussed ethics: ethics as defined by an Institutional Review Board (IRB), data quality as a measure of ethics, ethics as defined by theory, and social justice-minded ethics.

Building from the recommendations of the Belmont Report and Cain et al.’s (2019) review, we identified three questions related to ethics: (a) Is IRB or ethical board approval described? (b) Is informed consent and participant confidentiality described? (c) Is data quality described? Examples of data quality for quantitative strands included piloting the data collection instrument, use of a validated instrument (e.g., reviewed by experts), or addressing implementation bias (i.e., how the instrument was administered; Onwuegbuzie, 2003). Examples for qualitative strands included piloting the data collection instrument (e.g., interview protocol), member checks, triangulation, quotations or thick descriptions, or including a reflexivity statement of researcher bias (Cain et al., 2019; Creswell & Plano Clark, 2018). As seen in Table 8, ratings of ethics rigor were based on descriptions of various components across no, one or both strands. The expanded framework consists of these five questions (two paradigm and three ethics) and the initial 23 items.

Table 8.

Ethical Considerations.

	High	Medium-High	Medium	Low-Medium	Low
a. Is IRB or ethical board approval described?	Yes	Yes	Yes	No	No
b. Is informed consent and participant confidentiality described?	Yes—both strands	A. Both items for both strands B. At least one item for both strands C. none D. At least one item for one strand	No	At least one item from either b or c	No
c. Is data quality described?	Yes - both strands	A. none B. At least one item for one strand C. At least one item for both strands D. At least one item for both strands	No	At least one item from either b or c	No

Note. For medium-high rigor, the document must meet item a and (A) describe informed consent and confidentiality for both strands but not data quality; (B) describe informed consent OR confidentiality for both strands and data quality for one strand; (C) describe data quality for both strands but not informed consent or confidentiality; OR (D) describe informed consent and confidentiality for one strand and data quality for both strands.

Empirical Results

This section presents several empirical findings, including the rigor ratings for the two additional elements with examples, the expanded overall rigor score (i.e., the sum of the original overall rigor, paradigm rigor, and ethics rigor), and the regression analysis results.

The majority of the documents (89.39%) reported no information on a paradigm (Table 9). For the remaining 10.61% documents, the rigor ratings were either medium (4.55%), medium-high (1.52%), or high (4.55%). Only 7.58% documents explicitly identified a paradigm for the overall research while 3.03% identified a stance for one strand only. On average, the paradigm rigor score was 1.32 corresponding to low rigor. Walker (2017) and Parey (2020) are examples of high paradigm rigor which reported on a multiple paradigm stance and single stance, respectively. Importantly, both linked the paradigm to the methodological approach.

Table 9.

Count of Documents by Rigor Rating and Element.

Categories	Paradigm	Ethics	Overall (8 Elements)
Low	59	18	0
Low-medium	0	25	45
Medium	3	8	17
Medium-high	1	14	4
High	3	1	0
Total	66	66	66

For ethics consideration, the average rigor score was 2.32, corresponding to low-medium rigor. Only 1.52% of the documents fully described ethics (“high rigor”) while 33.33% were classified as medium (12.12%) or medium-high (21.21%; Table 9). The remaining 65.15% either had no information (low, 27.27%) or reported on permission from gatekeepers to carry out the research or other ethical considerations, but had no statement of IRB approval (low-medium, 37.88%). About 18% of the documents included data quality information for one strand but not both. None of the documents classified as having high or medium-high rigor in our methodological review displayed high ethics rigor. Fisher’s (2013) dissertation on teachers’ perceptions of working with children with specific disabilities reflected high ethics rigor. The researcher reported on approval from the IRB and gatekeepers, conducted a pilot study, and shared ethical considerations including debriefing about the study, its voluntary and confidential nature, and informed consent.

Regarding expanded overall rigor (Table 9), none of the 66 documents demonstrated high or low rigor. About 68.18% of the documents were rated as low-medium rigor with the remaining rated as medium (25.76%) or medium-high (6.06%). The mean expanded overall rigor was 19.65 (SD = 6.65), corresponding to low-medium rigor. For the regression model, the predictive power was approximately 90% (see Table 10). For paradigm, compared to documents with high rigor, documents rated as medium had 5.6 lower original overall rigor score (p = .025) and documents rated as low-medium had almost six points lower original overall rigor score (p < .001). There were no significant differences in original overall rigor for documents with high and medium-high paradigm rigor. Regarding ethics, documents with medium-high ethics rigor had about three points higher original overall rigor than documents with high ethics rigor (p < .001). Additionally, documents with low ethics rigor had approximately five points lower original overall rigor score (p < .001) than documents rated with high ethics rigor.

Table 10.

Regression Analysis Results.

Variable	Marginal Effects (Standard Errors)
Paradigm_medium-high	0.57 (0.96)
Paradigm_medium	−5.57 (2.48)**
Paradigm_low-medium	0
Paradigm_low	−5.97 (1.15)***
Ethics_medium-high	3.15 (1.19)***
Ethics_medium	0.85 (2.01)
Ethics_low-medium	−0.21 (0.58)
Ethics_low	−5.03 (0.97)***
Predictions within 5 points	59
Estimated predictive power (%)	89.40

Note. *p < 0.1, **p < 0.05, ***p < 0.01; n = 66; coefficients are marginal effects calculated at means.

Discussion

Mixed methods allow researchers to explore both breadth and depth of a phenomenon (Creswell & Plano Clark, 2018). Discussions of mixed methods research quality are increasingly prevalent, with authors from diverse fields identifying quality criteria that can be classified into common themes, although diversity among emphasized criteria also exists (Fàbregues & Molina-Azorín, 2017). While the Expanded Rigorous Mixed Methods Framework (ERMMF) focuses on rigor, given that rigor is the foundation for quality, the framework supports quality evaluations. Moreover, the primary elements and the aims and purpose element are aligned with Hirose and Creswell’s (2023) six core criteria for quality evaluation which build on the work of Bryman (2014) and consider standards from multiple stakeholders rather than solely mixed methods experts (Hirose & Creswell, 2023; Fabregues et al., 2019). Although omitted from their list, Hirose and Creswell (2023) acknowledge the importance of two other elements in quality evaluations: philosophical stance and incorporation of literature (criteria included in the ERMMF). Furthermore, mixed methods researchers have identified the importance of identifying philosophical assumptions and ethics when planning and undertaking mixed methods studies (Cain et al., 2019; Fàbregues & Molina-Azorín, 2017). With only eight elements, it could be argued that the ERMMF, consistent with the ideal, provides a limited number of quality criteria to guide researchers (Hirose & Creswell, 2023).

Furthermore, the ERMMF offers researchers a five-point continuum on which to evaluate rigor (i.e., reporting and methodological quality) which is distinct from other quality guidelines in the field. Specifically, while it could be argued that some tools could allow for a continuum of quality evaluation (e.g., GRAMMS’ ratings of “yes,” “yes, but…” and “no”; O’Cathain et al., 2008), these tools do not offer guidelines for each category on the continuum (Guetterman et al., 2023). Other tools rely on a dichotomous response (e.g., MMAT’s “yes” or “no”; Hong et al., 2018), which may not capture the nuances of applying mixed methods approaches. Unlike other guidelines, the ERMMF was originally designed to distinguish among varying levels of rigor and therefore may be a useful tool for researchers who are specifically interested in understanding different characteristics of rigor and quality reflected in a systematic methodological review.

The following sections discuss the contributions of the ERMMF to mixed methods research, practical implications, and limitations and future directions.

Contribution to Mixed Methods Research

Given the importance of quality evaluation in the field of mixed methods, the ERMMF makes three contributions to the field. First, in expanding the ERMMF, we explicitly clarified the relationship between rigor and quality. Specifically, using Hong and Pluye’s (2019) conceptualization of quality, we linked rigor with methodological and reporting quality and moreover, re-emphasized the concept of quality as encompassing conceptual, methodological and reporting quality (Hong & Pluye, 2019). Quality is therefore the conceptualization of the research phenomenon to generate new insights, the conduct of the study using scientifically accepted procedures, and reporting of the study. Furthermore, given the clear distinction between methodological and reporting quality when we rated an article as “medium-high” or “high” it reflected not only reporting quality but steps taken during the research process. Therefore, the five-category rigor continuum from low to high signifies a continuum from reporting quality to rigor (i.e., both methodological and reporting quality), with one being low reporting quality and five being high rigor.

Second, we operationalized an existing rigor tool by (a) expanding the continuum from three categories to five and (b) providing guiding questions for the original criteria. For the former, each element is operationalized for each category of the continuum. This is distinct from other guidelines which advocate for “appropriate” or “sufficient” demonstration of criteria, and the expansion of the five categories provided a more structured and nuanced approach to determining overall rigor. The guiding questions reduced the inconsistencies in rigor ratings when applying the framework. While a more open interpretation may arguably be aligned with the view that quality criteria should be flexible to facilitate application across different research contexts (Fàbregues & Molina-Azorín, 2017), in leaving criteria open to interpretation, novice researchers or interdisciplinary teams may find it challenging to consistently apply criteria. Therefore, the ERMMF may be especially useful to researchers who are new to mixed methods or for researchers collaborating across disciplinary or methodological boundaries. Moreover, including explicit criteria to evaluate methodological rigor, the ERMMF could serve as a guide to authors, researchers and dissertation supervisors when preparing manuscripts, and to reviewers and journal editors when evaluating the rigor of manuscripts.

Thirdly, our paper goes beyond conceptual arguments regarding the importance of ethics and paradigm by providing empirical evidence of their correlation with increased rigor using the case of inclusive education. According to the regression results, documents that provided no or limited information on paradigm or ethics had statistically significantly lower original overall rigor scores than documents with full descriptions. Additionally, paradigm and ethics elements had lower ratings as compared to the other six elements (i.e., low and low-medium, respectively) which suggests that reporting on paradigm and ethics is currently not highly prioritized. The literature on paradigm and ethics in mixed methods research is still evolving; however, more effort is required to promote the importance of these elements. Although the expanded framework is not a primer, it could guide researchers to incorporate paradigmatic and ethical considerations in their research.

Limitations and Future Directions

When operationalizing the elements, we maintained the criteria in Harrison et al.’s (2020) framework. The definition of mixed methods on which the framework is based might be a limitation for evaluating mixed methods studies. For example, Q-methodology, where quantitative and qualitative strands are not distinct, would not be considered a mixed methods approach as defined in Harrison et al. (2020). Future work should consider such approaches when constructing rigor criteria. Additionally, future research may investigate the effect of weighting primary elements more heavily than advanced elements and assess the framework’s interrater reliability and content validity. Descriptions of various elements might be potentially problematic due to journal word limits, but from the reviewed documents, many scholars have incorporated the information within the journal word limits, although the ethics element was not fully reflected in high-rigor articles. Journals should consider an increased word count for mixed methods studies and also continue to promote ethical standards in research. Given that (a) education is the discipline most represented in the mixed methods research quality literature (Fabregues et al., 2019), (b) approaches to appraising quality differs among researchers across disciplines (Fabregues et al., 2019), and (c) a goal of the original framework was its applicability across disciplines, applying the ERMMF to other disciplines is necessary to confirm the latter. Finally, although the ERMMF may support the work of researchers with varying methodological expertise and disciplinary orientations, some level of familiarity with mixed methods research approaches may be required to accurately apply the framework.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Bephyer Parey

References

Ainscow

Booth

Dyson

(2006). Improving schools, developing inclusion. Routledge.

Bryman

(2014). June 1989 and beyond: Julia Brannen’s contribution to mixed methods research. International Journal of Social Research Methods, 17(20), 121–131. https://doi.org/10.1080/13645579.2014.892653

Cain

L. K.

MacDonald

A. L.

Coker

J. M.

Velasco

J. C.

West

G. D.

(2019). Ethics and reflexivity in mixed methods research: An examination of current practices and a call for further discussion. International Journal of Multiple Research Approaches, 11(2), 144–155. https://doi.org/10.29034/ijmra.v11n2a2

Coombes

Chan

Allen

Foxcroft

D. R.

(2016). Mixed‐methods evaluation of the good behaviour game in English primary schools. Journal of Community & Applied Social Psychology, 26(5), 369–387. https://doi.org/10.1002/casp.2268

Corr

Snodgrass

M. R.

Love

Scott

I. M.

Kim

Andrews

(2020). Exploring the landscape of published mixed methods research in special education: A systematic review. Remedial and Special Education. https://doi.org/10.1177/0741932520924030

Creswell

J. W.

(2014). Research design (4th ed.). Sage.

Creswell

J. W.

Plano Clark

V. L.

(2018). Designing and conducting mixed methods research (3rd ed.). Sage.

Desutter

(2015). Utilizing novice teachers' perceptions and voices to make recommendations for improving teacher preparation for inclusive education: A mixed methods study. [Doctoral dissertation, University of North Dakota. https://search-proquest-com.ezproxy.uws.edu.au/docview/1700208780?pq-origsite=primo

Engelbrecht

Savolainen

(2018). A mixed-methods approach to developing an understanding of teachers’ attitudes and their enactment of inclusive education. European Journal of Special Needs Education, 33(5), 660–676. https://doi.org/10.1080/08856257.2017.1410327

10.

Fàbregues

Molina-Azorín

J. F.

(2017). Addressing quality in mixed methods research: A review and recommendations for a future agenda. Quality & Quantity, 51(6), 2847–2863. https://doi.org/10.1007/s11135-016-0449-4, https://link.springer.com/article/10.1007/s11135-016-0449-4

11.

Fàbregues

Paré

M. H.

Meneses

(2019). Operationalizing and conceptualizing quality in mixed methods research: A multiple case study of the disciplines of education, nursing, psychology, and sociology. Journal of Mixed Methods Research, 13(4), 424–445. https://doi.org/10.1177/1558689817751774

12.

Fisher

K. G.

(2013). Teacher perceptions of working with children with specific special education exceptionalities in the general education classroom. [Doctoral dissertation, University of Southern Mississippi]. Aquila Digital Community https://aquila.usm.edu/dissertations/397

13.

Greene

J. C.

Caracelli

V. J.

Graham

W. F.

(1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274. https://doi.org/10.3102/01623737011003255

14.

Guetterman

T. C.

Fetters

M. D.

Creswell

J. W.

(2015). Integrating quantitative and qualitative results in health science mixed methods research through joint displays. Annals of Family Medicine, 13(6), 554–561. https://doi.org/10.1370/afm.1865

15.

Guetterman

T. C.

Molina-Azorin

J. F.

Fàbregues

(2023). The need to rigorously develop common quality guidelines for reporting mixed methods research. Journal of Mixed Methods Research, 17(1), 6–11. https://doi.org/10.1177/15586898221143

16.

Hall

R. F.

(2013). Mixed methods: In search of a paradigm. In Le

(Eds), Conducting research in a changing and challenging world. Nova Science Publishers.

17.

Harrison

R. L.

Reilly

T. M.

Creswell

J. W.

(2020). Methodological rigor in mixed methods: An application in management studies. Journal of Mixed Methods Research, 14(4), 473–495. https://doi.org/10.1177/1558689819900585

18.

Hesse-Biber

S. N.

(2010). Mixed methods research: Merging theory with practice. Guilford Press.

19.

Hirose

Creswell

J. W.

(2023). Applying core quality criteria of mixed methods research to an empirical study. Journal of Mixed Methods Research, 17(1), 12–28. https://doi.org/10.1177/15586898221086346

20.

Hong

Q. N.

Fàbregues

Bartlett

Boardman

Cargo

Dagenais

Gagnon

M.-P.

Griffith

Nicolah

O’Cathain

Rousseau

M.-C.

Vedel

Pluye

(2018). Mixed methods appraisal tool (MMAT), version 2018 for information professionals and researchers. Education for Information, 34(4), 285–291. https://doi.org/10.3233/efi-180221

21.

Hong

Q. N.

Pluye

(2019). A conceptual framework for critical appraisal in systematic mixed studies reviews. Journal of Mixed Methods Research, 13(4), 446–460. https://doi.org/10.1177%2F1558689818770058

22.

Howell Smith

M. C.

Shanahan Bazis

(2021). Conducting mixed methods research systematic methodological reviews: A review of practice and recommendations. Journal of Mixed Methods Research, 15(4), 546–566. https://doi.org/10.1177/1558689820967626

23.

International Institute For educational planning . (2020). Disability inclusive education and learning (Brief 3). United Nations Educational, Scientific and Cultural Organization. https://learningportal.iiep.unesco.org/en/issue-briefs/improve-learning/learners-and-support-structures/disability-inclusive-education-and

24.

Kutscher

Parey

(2022). A scoping review of mixed methods rigour in inclusive education: application of the Rigorous Mixed Methods Framework. International Journal of Inclusive Education, 1–16. https://doi.org/10.1080/13603116.2022.2100491

25.

Leech

N. L.

(2012). Writing mixed research reports. American Behavioral Scientist, 56(6), 866–881. https://doi.org/10.1177%2F0002764211433800

26.

Leech

N. L.

Onwuegbuzie

A. J.

(2010). Guidelines for conducting and reporting mixed research in the field of counseling and beyond. Journal of Counseling & Development, 88(1), 61–69. https://doi.org/10.1002/j.1556-6678.2010.tb00151.x

27.

Levitt

H. M.

Bamberg

Creswell

J. W.

Frost

D. M.

Josselson

Suárez-Orozco

(2018). Journal article reporting standards for qualitative primary, qualitative meta-analytic, and mixed methods research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73(1), 26–46. https://doi.org/10.1037/amp0000151

28.

Molina-Azorin

J. F.

Fetters

M. D.

(2016). Mixed methods research prevalence studies: Field-specific studies on the state of the art of mixed methods research. Journal of Mixed Methods Research, 10(2), 123–128. https://doi.org/10.1177/1558689816636707

29.

Parey

(2020). Accommodations for the inclusion of children with disabilities in regular schools in Trinidad: a mixed methods approach. International Journal of Inclusive Education, 26(6), 559–575. https://doi.org/10.1080/13603116.2019.1701719

30.

O’Cathain

(2009). Reporting mixed methods projects. In Andrew

Halcomb

E. J.

(Eds), Mixed methods research for nursing and the health sciences. Wiley/Blackwell.

31.

O'Cathain

Murphy

Nicholl

(2008). The quality of mixed methods studies in health services research. Journal of Health Services Research & Policy, 13(2), 92–98. https://doi.org/10.1258%2Fjhsrp.2007.007074

32.

Onwuegbuzie

A. J.

(2003). Expanding the framework of internal and external validity in quantitative research. Research In The Schools, 10(1), 71–89.

33.

Onwuegbuzie

A. J.

Corrigan

J. A.

(2018). What is happening now? An overview of mixed methods applications in special education. Research in the Schools, 25(2), 1–22.

34.

Pluye

Hong

Q. N.

(2014). Combining the power of stories and the power of numbers: Mixed methods research and mixed studies reviews. Annual Review of Public Health, 35(1), 29–45. https://doi.org/10.1146/annurev-publhealth-032013-182440

35.

Poth

C. N.

(2018). Innovation in mixed methods research. Sage.

36.

Sale

J. E.

Brazil

(2004). A strategy to identify critical appraisal criteria for primary mixed-method studies. Quality and Quantity, 38(4), 351–365. https://doi.org/10.1023/b:ququ.0000043126.25329.85

37.

Shannon-Baker

(2016). Making paradigms meaningful in mixed methods research. Journal of Mixed Methods Research, 10(4), 319–334. https://doi.org/10.1177/1558689815575861

38.

Stadnick

N. A.

Poth

C. N.

Guetterman

T. C.

Gallo

J. J.

(2021). Advancing discussion of ethics in mixed methods health services research. BMC Health Services Research, 21(1), 577. https://doi.org/10.1186/s12913-021-06583-1

39.

Teddlie

Tashakkori

(2009). Foundations of mixed methods research. Sage.

40.

Walker

(2017). Inclusion or illusion: A mixed methods study of pedagogical practices to enhance pre-service teachers' preparedness for contemporary inclusive teaching. Western Sydney University. [Unpublished doctoral dissertation].

41.

Wisdom

J. P.

Cavaleri

M. A.

Onwuegbuzie

A. J.

Green

C. A.

(2012). Methodological reporting in qualitative, quantitative, and mixed methods health services research articles. Health Services Research, 47(2), 721–745. https://doi.org/10.1111/j.1475-6773.2011.01344.x

42.

Wood

(2017). Secondary special educators' attitudes and sense of self-efficacy toward inclusive education. San Diego: [Doctoral dissertation, University of California.

43.

Wooldridge

J. M.

(2002). Econometric analysis of cross section and panel data. MIT Press.