Advancing Quality Standards in Mixed Methods Research: Extending the Legitimation Typology

Abstract

Several scholars have proposed frameworks for assessing the quality of mixed methods research (MMR) studies. However, no general consensus has emerged. The legitimation typology developed by Onwuegbuzie and Johnson (2006) is one promising approach that addresses quantitative, qualitative, and MMR elements. The aim of this intrinsic, exploratory case study is to explore the use of the legitimation typology in empirical MMR studies and through interviews with the developers, MMR scholars, and researchers who applied the legitimation typology to an empirical MMR study. We conducted a systematic methodological review using multiple databases and identified 49 empirical MMR studies that addressed the legitimation typology. Using a critical case sampling approach defined by participants’ unique experiences with the legitimation typology, semi-structured interviews were conducted with five authors of empirical MMR studies, a mixed methods researcher who has written about the typology, and one of the authors of the original legitimation typology to expand on ways the legitimation typology is used in practice. Four overarching themes were identified: (a) comprehensive approach to assessing quality, (b) researchers’ interpretation of legitimation types, (c) value of divergent findings, and (d) strategies for applying the legitimation typology. This case study adds to the MMR literature by clarifying the use of emic-etic and conversion legitimations and by proposing a new legitimation type: divergent findings legitimation. Hence, this study elucidates the application of one quality framework (i.e., legitimation) in MMR and provides recommendations to the field to further advance discussions on quality criteria and their implementation in mixed methods research.

Keywords

Legitimation quality validity mixed methods research

Determining what represents quality in mixed methods research (MMR) is an ongoing subject of debate that holds important implications for the conduct and evaluation of MMR. The assessment of quality in MMR studies is influenced by the interpretation of different audiences such as academic journals, funding agencies, publishers, and professors (Plano Clark & Ivankova, 2016), adding further complexity to an already intricate process. Without generally agreed-upon procedures to assess quality in MMR studies, determining the credibility and transferability of meta-inferences that integrate quantitative and qualitative components is challenging (Collins et al., 2012), thus limiting the potential contributions of MMR studies.

Fàbregues and Molina-Azorín (2017) conducted a systematic methodological literature review to examine quality frameworks used in MMR. In analyzing 35 different quality frameworks, they identified 19 quality criteria across four distinct study phases (i.e., planning, undertaking, interpreting, and disseminating/applying findings). They noted most quality criteria included the importance of each strand standing on its own, developing clear aims and research questions, and establishing consistencies between the meta-inferences and the findings. Fàbregues and Molina-Azorín (2017) concluded that more empirical research on quality in MMR studies, greater consistency in terms used to describe quality in MMR, and agreement on the quality indicators for MMR are needed.

Our study responds to Fàbregues and Molina-Azorín’s (2017) explicit call for “future studies that take into account researchers’ views on how quality should be conceptualized and operationalized, as this would not only improve the usefulness and applicability of existing quality frameworks but would also lead to the inclusion of criteria and other aspects not taken into account to date” (p. 2858). The legitimation typology proposed by Onwuegbuzie and Johnson (2006) and refined by Johnson and Christensen (2017, 2020) addresses the quality of MMR studies through 11 potential validity threats. The legitimation typology is both holistic and synergistic in that it incorporates major works of quality in MMR and follows four core principles to accomplish synergy (Collins et al., 2012). Moreover, the legitimation typology follows Greene’s (2006) four methodological domains for establishing a methodological or research paradigm (Onwuegbuzie, Johnson et al., 2011).

We selected the legitimation typology as the focus of our study for several reasons. First, it is widely written about by MMR scholars and referenced in MMR texts. Second, this typology uses distinctive MMR terms, a “bilingual nomenclature” (Onwuegbuzie & Johnson, 2006, p. 48), that minimize potential quantitative or qualitative biases. Hence, it contributes to Fàbregues and Molina-Azorín’s (2017) call for greater consistency in terminology. Finally, based on our own experiences teaching and conducting MMR, we believe that the legitimation typology holds the most potential of all frameworks proposed to date for effectively assessing quality in MMR. Therefore, the purpose of this intrinsic, exploratory case study is to (a) examine how researchers are using the legitimation typology when conducting MMR and (b) extend and refine the legitimation typology to better realize its potential for use in the field of MMR. Our study is guided by the following research questions:

How are researchers using the legitimation typology to address quality in MMR studies?

What are the experiences of researchers who have used the legitimation typology to address quality in MMR studies?

How can legitimation concepts be further elaborated to support their broader application in MMR studies?

The methodological contribution of this study is the empirical evidence of how mixed methods researchers have used the legitimation typology and resultant recommendations to guide its use in future studies.

Method

We employed an intrinsic, exploratory case study design that focused on the study of the case itself viewed as a unique bounded system (Stake, 1995). The bounded system was defined as the use of the legitimation typology in an empirical MMR study by mixed methods researchers from 2006 to 2020. This case study was exploratory as we explored this topic in greater detail with no pre-established outcomes, and intrinsic as it focused on the case and its features from the perspective of different mixed methods researchers. Our study consisted of two data sources: (a) empirical studies employing the legitimation typology, and (b) interviews with MMR scholars who wrote about the typology and researchers who applied it in their empirical studies. The recommendations provided are based on our case analysis.

MMR Systematic Methodological Review

We conducted an MMR systematic methodological review (MMR-SMR) (Howell Smith and Shanahan Bazis, 2020) using a multi-phase search to identify empirical MMR studies that explicitly addressed one or more legitimation type(s). Phase I round 1 involved a search for potential studies between 2006 and 2017 using PsycINFO, Web of Science, and the Journal of Mixed Methods Research (JMMR) using search terms such as mixed method*, mixed-method*, mixed research, quantitative AND qualitative, inference, legitimation, quality, validity, credibility, and transferability. We eliminated duplicate records and removed entries that were not published in English. We reviewed the remaining abstracts for evidence of an empirical peer-reviewed mixed methods study. To determine the final eligibility of the remaining studies, we reviewed the full text for evidence that the article addressed at least one form of legitimation by conducting a lexical search for the word “legitimation,” and reviewing the references for inclusion of Onwuegbuzie and Johnson (2006) or Johnson and Christensen (2017). Finally, we read the full text and retained studies that explicitly referred to at least one legitimation type AND described how that type was addressed in the study. To ensure reliability, three members of the research team coded all articles for inclusion/exclusion criteria and came together to discuss discrepancies. Using this process, we identified a total of 13 MMR studies that addressed legitimation in round 1. Due to the small number of studies identified in Phase I, round 1, we repeated these search procedures using the same databases and extended the years from 2013 to 2019. We identified a total of 12 additional MMR studies that addressed legitimation in Phase I, round 2.

For Phase II, we used Google Scholar’s “cited by” feature to identify articles citing the original legitimation typology article (Onwuegbuzie and Johnson 2006). Although the search results indicated 1,687 records, Google Scholar provided access to the first 1,000 results (Haddaway et al., 2015). We downloaded these citations for review. To conduct a more refined search within the pool of 1,687 records, we used the ‘search within citing articles’ feature in Google Scholar with the keyword “legitimation” to only include records explicitly mentioning the legitimation typology. This yielded 542 entries of which we were able to access and downloaded 530 citations, 346 of which were duplicates in the pool of 1,000 citations we previously downloaded. This resulted in a total of 1,184 potential studies. Using the same screening and eligibility review steps described in Phase I, we identified 46 MMR studies that addressed legitimation in Phase II between 2006 and 2020.

As a final check for completeness, we conducted a second-round search using Google Scholar’s “cited by” feature for articles that cited Johnson and Christensen (2017, 2020). Google Scholar indicated both texts were cited 13,447 times. We filtered the search using the term “legitimation,” resulting in 348 citations for both searches. A final filter was applied for citations since 2017, and both searches returned 161 citations. We downloaded 161 records for the search based on the most recent edition of the text. We used the same screening and eligibility review steps described in previous rounds and eliminated all 161 records from the Phase II, round 2.

Each phase of the identification process was conducted independently; therefore, we compared the results from Phase I (n = 25) and Phase II (n = 46) and identified 22 duplicates. Thus, our final pool included 49 studies. The four rounds of searches using two very different approaches yielded a robust pool of studies that informed our research questions. The PRISMA diagram (Figure 1) details the reasons and frequencies for excluding records across all phases and rounds of the MMR-SMR.

Figure 1.

PRISMA diagram of literature search and review procedures.

Qualitative Interviews

We used a critical case sample to identify participants across three groups: authors who developed the legitimation typology, authors who have discussed the legitimation typology, and authors of empirical MMR studies that used the legitimation typology identified from our searches. Critical case sampling was most appropriate as participants were divided into groups based on their unique experiences using the legitimation typology. We contacted 13 authors identified from Phase I, round 1 of the MMR-SMR and five accepted to participate in the study. We also contacted six MMR scholars and two agreed to participate, the coauthor of the legitimation typology, Johnson, and Ivankova. We also incorporated examples from articles gathered from our MMR-SMR authored by Onwuegbuzie, the author of the legitimation typology, through inclusion of 12 published empirical studies that addressed legitimation. This emic perspective (Onwuegbuzie, 2012) allowed us to more broadly explore the legitimation typology based on Onwuegbuzie’s publications and maximize interactions through document analysis. The total sample size included seven participants, sufficient to reach data saturation (Guest et al., 2006).

All participants agreed to be identified in this study, which was approved by the Institutional Review Board at the first author’s institution. The first author conducted all interviews via Skype using a semi-structured interview protocol and transcribed all interviews. Interview questions included a mixture of basic descriptive questions (e.g., Discuss the importance of validity in mixed methods research.), follow-up/clarifying questions (e.g., Please explain how you addressed [specific legitimation] in your study.), experience/example questions (e.g., How have your views of the legitimation typology changed and is there anything you would like to change about the current typology?), and closing questions (e.g., Is there anything else you would like to add?) (Janesick, 2016). Table 1 presents participant information and their experience with the legitimation typology.

Table 1.

Participants and Experience with Legitimation Typology.

Name	Experience with legitimation typology
Burke Johnson	Johnson developed the original legitimation typology with Anthony Onwuegbuzie. Johnson has continued to develop the legitimation typology, writing about it in several articles and books.
Nataliya Ivankova	Ivankova addressed legitimation in two studies. Kawamura et al. (2009) addressed legitimation in a study of parasocial interaction of an entertainment–education program audience. Torres-Alzate et al. (2020) addressed legitimation in a Delphi study of global health competencies for baccalaureate nursing students in the United States. Ivankova has also written about legitimation in several articles and books.
Judith Schoonenboom	Schoonenboom (2016) addressed legitimation in a study using multilevel mixed intact group analysis (MMIGA), where differences among intact subgroups are first quantitatively identified and subsequently qualitatively described and explained by differences among the contexts of separate intact subgroups. Schoonenboom has also written about legitimation in several articles and books.
Carolina Bustamante	Bustamante (2019) addressed legitimation in a study to understand the impact of a professional development program on Web 2.0 technologies for teachers of Spanish by understanding teachers’ experiences and their outcome measures on the program.
Joseph Cooper	Cooper and Hall (2016) addressed legitimation in a study to understand Black male student athlete’s experiences at a Black college/university using questionnaire data and interviews.
Sherry Dismuke	Martin and Dismuke (2018) addressed legitimation in a study to examine writing teacher practices of in-service and preservice elementary school teachers using complexity theory.
David Swedler	Swedler et al. (2015) addressed legitimation in a study guided by the Theory of Planned Behavior to examine how personal and workplace factors affect truck drivers’ decision-making when engaging in distracted driving.

To analyze the interviews, we engaged in two cycles of coding. In the first cycle, we identified in-vivo and initial codes across all interviews following Merriam’s (1988) case study guidelines for the initial development of categories. We engaged in a second cycle of coding by developing categories based on initial codes. Once categories for each individual interview were created, we incorporated pattern coding (Saldaña, 2021) to group summaries or codes into smaller categories or themes. Similar codes were grouped by categories, and we found concordance by matching similar categories across all participant interviews to develop patterns. These patterns became the overarching themes of the study. All interviews were coded using MAXQDA (VERBI Software, 2017). To ensure reliability and trustworthiness, member-checking interviews were conducted with all participants. Interview summaries were read verbatim to participants and asked to comment on their accuracy. Changes and additions were incorporated to varying degrees across all participant interviews. Memo notes and audit trails were used to further assess the trustworthiness of the data.

Findings

Mixed Methods Research Systematic Methodological Review

We identified 49 studies that cited the original Onwuegbuzie and Johnson (2006) legitimation article, explicitly addressed at least one type of legitimation, and described how they addressed it. Only two (11.1%) cited a newer iteration of the legitimation typology (e.g., Johnson & Christensen, 2017). As noted in Figure 2, the first study was published in 2007 with the number of studies generally increasing over time. Six (12%) of the studies were published in JMMR and four (8%) were published in the International Journal of Multiple Research Approaches, both MMR-focused publications.

Figure 2.

Number of empirical MMR studies that addressed legitimation by year.

Of these 49 studies, 33 (67%) were related to the field of education, seven (14%) to health, six (12%) to both education and health, and three (6%) to social science. Twenty-four (49%) studies used a convergent design, 17 (35%) used an exploratory design, seven (14%) used an explanatory design, and one (2%) used complex applications across multiple phases of a study. On average, each study used 4.50 (SD = 2.97) legitimation types, ranging from one to ten types in each study. Table 2 summarizes how many studies used each legitimation type, with sample integration and emic-etic being the most common types used. Table 3 provides a description and illustration for each legitimation type.

Table 2.

Frequency of Legitimation Types Used in Empirical MMR Studies.

Legitimation Type	Frequency	Percentage (%)
Sample integration	33	67
Emic-etic	32	65
Weakness minimization	30	61
Multiple validities	29	59
Commensurability approximation	24	49
Conversion	20	41
Paradigmatic mixing	20	41
Multiple stakeholders	15	31
Sequential	15	31
Integration*	2	4
Pragmatic*	2	4

Note. N = 49; *indicates new legitimation type introduced in 2017.

Table 3.

Legitimation Types and Examples.

Legitimation Type	Description	How legitimation type was addressed
Sample integration	The degree to which appropriate conclusions are made from the quantitative and qualitative samples and appropriately integrated to produce quality meta-inferences.	Bustamante (2019) used the same sample in the qualitative and quantitative strands.
Emic-etic	The degree to which a researcher accurately depicts the insider’s (emic) and outsider’s (etic) perspectives in understanding or explaining a phenomenon.	Martin and Dismuke (2018) used peer debriefings and member checks with both insider participants and multiple outside researchers.
Weakness minimization	The degree to which weaknesses of one method are augmented by the strengths of another method in the study to realize the potential of MMR.	Howell Smith et al. (2019) used an exploratory design to maximize the strengths of an initial grounded theory strand to develop an instrument that was tested in the quantitative strand.
Multiple validities	The degree to which a researcher uses an appropriate subset or combination of the pertinent quantitative, qualitative, and mixed methods “validities” for a particular research study, to produce defensible/justified findings and high-quality meta-inferences.	Torres-Alzate et al. (2020) ensured the content validity of the quantitative strand, applied verification strategies such as team audits for the qualitative strand, and assessed the full mixed methods study using the legitimation typology.
Commensurability approximation	The degree to which the researcher makes cognitive switches from quantitative to qualitative lenses to yield syntheses and high-quality meta-inferences and a more nuanced worldview.	Onwuegbuzie et al. (2014) provided a third well-informed viewpoint by incorporating diverse perspectives from different members in the research team.
Conversion	The degree to which quantitizing and qualitizing data produced interpretable results and high meta-inference quality.	Cooper and Hall (2016) used an extensive codebook with quantized codes to increase transparency of the conversion process.
Paradigmatic/Philosophical	The degree to which a researcher successfully integrates their epistemological, ontological, axiological, methodological, and rhetorical beliefs underlying the quantitative and qualitative approaches into a workable mixed methods paradigm or research approach.	Swedler et al. (2015) used the independent analysis of the quantitative and qualitative strands to address the value of each strand through paradigmatic mixing.
Multiple stakeholder	The degree to which the researcher has addressed the interests of different stakeholders (e.g., quantitative, qualitative, and MM researchers, policymakers, groups with little power) in the report and generation of high-quality meta-inferences.	Goodwin et al. (2013) held dissemination and discussion meetings with group representatives, members of the media, professional associations, and other participants from the investigated community.
Sequential	The degree to which a later stage in a sequential design appropriately builds on an earlier stage, and researchers have addressed whether and how findings might have been different if the phases in the sequence were reversed.	Smith and Gillespie (2016) intentionally gathered survey data first to expose participants to consistent definitions to develop a shared understanding of their phenomenon before conducting interviews.
Integration*	The degree to which the researcher has successfully integrated quantitative and qualitative data, analysis, and conclusions.	Younas et al. (2020) used joint displays to maximize integration legitimation and building and merging techniques in their study.
Pragmatic*	The degree to which the purpose and goals of the research were met, the research problem was solved, the research questions were answered, and “actional results” were provided.	Liu et al. (2019) met their research goal of exploring the impact of a health education intervention by verifying the pattern of behavior changes and the influences on these changes that were demonstrated within the findings.

Note. Descriptions based on Johnson and Christensen (2017, 2020); Onwuegbuzie and Johnson (2006).

Additionally, we coded articles for the length of description provided about legitimation (i.e., one paragraph or fewer [≤1¶]; more than one paragraph [>1¶]) and whether the article included a legitimation table. We found that increased description lengths and inclusion of a table trended with a higher number of legitimation types addressed (see Table 4).

Table 4.

Average Number of Legitimation Types by Legitimation Description Length.

Description lengths	Articles (n)	Mean # of legitimation types (SD)
One paragraph or fewer	18	2.28 (1.96)
More than one paragraph	18	4.39 (2.03)
More than one paragraph+table	13	7.85 (2.08)
Total	49	4.50 (2.97)

Qualitative Interviews

Four overarching themes were identified from the interviews: (a) comprehensive approach to assessing quality, (b) researchers’ interpretation of legitimation types, (c) value of divergent findings, and (d) strategies for applying the legitimation typology. Descriptions and examples of each theme are presented in the following sections.

Comprehensive Approach to Assessing Quality

Understanding why researchers selected the legitimation typology is important to contextualize how they used it in practice. Participants universally endorsed the legitimation typology due to its comprehensive coverage of MMR quality indicators. As one of the authors of the original typology, Johnson, emphasized, “the legitimation framework is for validating the full mixed methods study, not just each strand.” The holistic approach resonated with Cooper who connected with the typology’s focus on viewing paradigms and methods on a continuum. He recalled that Onwuegbuzie and Johnson (2006) encouraged scholars to “not look at the different paradigms and methods as dichotomous, but rather on a continuum.” This encompassing perspective aligned with his existing views of qualitative and quantitative research, as he said, “I don’t think these are mutually exclusive.”Ivankova noted the legitimation typology’s comprehensiveness of particular value for helping students conceptualize MMR quality indicators. She also noted that the term legitimation is unique to MMR and contributes to the field’s growing vocabulary. Dismuke mentioned that she wanted to stay true to mixed methods, which meant using a framework to assess quality specific to a mixed methods design and addressing the legitimation types pertinent to their study. Bustamante selected the legitimation typology because there were several elements that were directly relevant to her convergent MMR design, such as sample integration and multiple validities. She explained that in “the data collection for a true convergent design, the same sample has to be used from both sides to make the data merge at the end.”Schoonenboom mentioned that she used the legitimation typology because it is widely known and specifically relevant to MMR. However, her purpose for using the legitimation typology was to investigate how each legitimation type applied to the mixed methodology she developed, rather than to the empirical study that she used as an example.

Summary

Overall, participants used the legitimation typology due to its comprehensiveness. The legitimation typology not only incorporates elements for assessing the validity and trustworthiness of the quantitative and qualitative study, respectively, but the full mixed methods study. The legitimation typology incorporates distinct MMR terminology and covers multiple mixed methods characteristics, encouraging its applicability across various mixed methods designs.

Participants’ Interpretations of Legitimation Types

Although participants selected the legitimation typology due to its comprehensive approach, its application required interpretation on their part for how it applied to their studies. In the following sections, we present illustrations and explanations for each of the 11 legitimation types. Johnson provided additional explanations and clarifications of the legitimation types, some of which were described in the updated typology (Johnson & Christensen, 2017, 2020). It is important to note that except for Johnson, participant comments were based on the original typology (Onwuegbuzie & Johnson, 2006), and therefore did not address pragmatic and integration legitimations that first appeared in 2017. To support the most recent conceptualizations of the legitimation typology, we refer to each type by its current name (2020), referring to the original (2006) and revised (2017) names as appropriate.

Sample Integration Legitimation

Cooper mentioned this legitimation type did not fit his study as he described the “quantitative data did not fit true positivistic criteria” such as random sampling or non-probability sampling. He described a positivistic framework as one where consensus is reached on what is deemed acceptable and finding truth. In his study, he used criterion sampling, focusing on a particular institution to identify participants that met the study’s criteria. His goal was not to generalize these results or transfer to other populations, but to identify participants. Alternatively, Martin and Dismuke (2018) addressed this legitimation type by ensuring the same sample was used for their quantitative and qualitative strands. Dismuke added that even though they had a small sample size, the fact that the same sample was used in both studies allowed them to triangulate data from multiple sources. Similarly, Onwuegbuzie, Frels et al. (2011) collected data using the same group of participants for the quantitative and qualitative strands to address sample integration legitimation.

Emic-Etic Legitimation

Previously known as inside-outside legitimation, Johnson mentioned the name was changed to clearly associate with terms historically used in ethnographic research and known more widely in MMR. Onwuegbuzie et al. (2007) described amplifying student perceptions of effective characteristics of college instructors as the insider’s (i.e., emic) view and compared these with their perceptions on a teaching evaluation form as the outsider (i.e., etic) view.

Cooper expressed one challenge of this legitimation type is determining how much data constitutes the insider and outsider perspectives. He explained that he did not address inside-outside legitimation because although the participants are the “insiders” and he used in-vivo codes and rich descriptions, the study is “still very much my outsider extraction of what they told me and my interpretation after analyzing the data from their perspectives… so it’s kind of like an outsider presenting the insider’s perspectives as opposed to if it was a true insider-outsider.” To balance the emic and etic perspectives, Cooper suggested that researchers should become familiar with “the sample, the research setting, the nature of the research questions, and outsiders.”

Weakness Minimization Legitimation

Cooper provided an example of this noting that qualitative research typically involves smaller sample sizes, whereas quantitative research can “provide more voices.”Schoonenboom agreed that both the quantitative and qualitative strands have specific weaknesses of their own that are compensated by the strengths of the other. She suggested that researchers may be able to reduce the limitations of one method by implementing an additional data collection method, such as observations. Nonetheless, Schoonenboom cautioned this will not apply in all cases and ultimately depends on the study characteristics and a researcher’s judgment on the best way to minimize weaknesses and increase the strengths of their study. Across nine studies conducted by Onwuegbuzie (e.g., Onwuegbuzie et al., 2014, 2007; Onwuegbuzie, Frels et al., 2011), weakness minimization was addressed by combining descriptive precision (i.e., qualitative analyses) with empirical precision (i.e., quantitative analyses).

Multiple Validities Legitimation

Meta-inferences integrate inferences from the quantitative and qualitative strand to develop overarching mixed methods inferences based on the full mixed methods study, whereas inferences are conclusions derived from a single methodology. Johnson asserted that this is the most important legitimation type; “the parts as well as the whole have to lead to good inferences, have to be of high quality.” He explained that for the MMR study to be legitimate, the quantitative strand must be valid, the qualitative strand must be trustworthy, and the synthesis must be legitimate. Johnson recommends considering the QUAN validity, QUAL trustworthiness, and mixed methods legitimation types when planning, designing, and conducting the study because it is too late to fix any deficiencies after the study has been conducted. To reinforce Johnson’s statement, Onwuegbuzie et al. (2007) optimized participant enrichment, instrument fidelity, and significance enrichment, in addition to inter-rater reliability, member-checking, and debriefing, to address multiple validities legitimation threats.

Ivankova discussed the importance of assessing validity and trustworthiness of each individual strand and its implications if not done properly. She noted, “if you have flaws, if you don’t have reliable or strong inferences coming from your quantitative and qualitative [strands], then your overall mixed methods inferences will probably also have flaws.” To ensure multiple validities in his study, Cooper described providing evidence of content validity and emerging construct validity of questionnaire data and “a whole range of things to ensure credibility and trustworthiness like member checks, subjectivity statements, detailed memos, audit trails, detailed descriptions, and rich data” of the qualitative data. Swedler acknowledged the importance of multiple validities. He stated, “if it wasn’t for having both methods, I wouldn’t have been able to really draw [meta-]inferences.”Swedler encouraged amplifying the value of what could be considered two individual studies, consistent with Fetters and Freshwater’s (2015) concept of 1+1=3. He described, “I want to make sure I’m getting more than just the sum of the parts…I want to make sure I can go beyond just the value of each individual one.”

Commensurability Approximation Legitimation

Johnson used the metaphor of putting on different glasses (quantitative, qualitative, and mixed) to describe the idea of switching perspectives. By continually moving back and forth, iteratively viewing a project through different lenses, researchers develop a “mixed viewpoint.” These switches yield a deeper understanding of the different approaches that ultimately contribute to a “larger understanding from those differences.” Sometimes these switches occur within the perspectives of a single researcher, but they can also occur among members of a research team. Johnson noted that “if you have a qualitative person and a quantitative person and then, say you’re going to be the mixed person, then the commensurability would be the agreed-upon synthesis from [the research team] or the meta-inferences you get from them that [the research team] say are legitimate.” Therefore, commensurability approximation focuses on the syntheses of the differences, or how different ideas come together from different sources in a study. Originally called commensurability, Johnson explained that the name was changed from commensurability legitimation to commensurability approximation legitimation because full commensurability across paradigms cannot be fully achieved; nonetheless, it should be a goal to work toward. Onwuegbuzie, Frels et al. (2011) incorporated a diverse team of researchers with respect to research orientation (e.g., quantitative, qualitative, and mixed method), college teaching experience (e.g., assistant, associate, and full professors), and discipline (e.g., special education, educational assessment, research methodologist, and teacher educator).

In general, commensurability approximation was not seen as relevant by most participants. Cooper described that “commensurability allowed for the latitude for me to say, ‘this is a qualitative dominant mixed methods study that is using quantitative data as complementary data to further crystalize and explain or explore a particular phenomenon of interest.’”Schoonenboom explained that commensurability legitimation was not relevant to her MMIGA model. She explained that the purpose of the quantitative strand was to identify a particular group of interest, and therefore, the inferences were based only on the qualitative study that followed. Hence, there was not a need to switch between differing perspectives. However, Schoonenboom recommends researchers engage in some form of perspective switching to develop meta-inferences and including examples on how to do this would be helpful.

Conversion Legitimation

Some participants discussed potential issues that can arise when quantitizing and qualitizing data. For instance, Cooper noted that just because something was infrequent in the qualitative data does not mean it is unimportant. He illustrated an example from his study with two focus groups—one provided scant information about a particular subject in comparison to the other, which discussed the subject in greater depth. Cooper verified the importance of this subject through member-checks and peer debriefings. He also recommended researchers verify data transformation through crystallization and triangulation.

Dismuke described the challenge of determining how to analyze converted data. Although Dismuke and colleagues used frequencies and created narratives to quantitize and qualitize data, they did not address conversion legitimation in the published article due to space constraints. Dismuke believed, “if I was ever going to actually include it [conversion legitimation] in the article, I would need a lot more information on how to weigh it and how to actually do a statistical analysis in this way… I almost felt like I needed an entire class in that, if that makes sense, to figure out the statistical part on how to do it.” Onwuegbuzie, Frels et al. (2011) addressed conversion legitimation by verifying quantitative analyses using correspondence analysis via member-checks.

Paradigmatic/Philosophical Legitimation

Participants offered varying interpretations of paradigmatic legitimation. Cooper expressed that based on his interpretation, paradigmatic legitimation occurs when both the quantitative and qualitative strands are equally weighed in the study “within the level of inquiry.” He believed his study needed a more rigorous quantitative component to balance the contributions of his qualitative strand to obtain paradigmatic legitimation.

Schoonenboom’s interpretation of paradigmatic mixing focused on worldviews. She explained that simply because a researcher is conducting a quantitative study, it should not dictate that they have a fully objective view, nor does it imply that a researcher will have a fully subjective viewpoint for qualitative study. She did not believe researchers will have different beliefs depending upon their research methodology. Nevertheless, she acknowledged that several different perspectives and worldviews can emerge when working in research teams, but these might not necessarily be connected to the quantitative or qualitative strands. Onwuegbuzie et al. (2007) addressed paradigmatic legitimation by implementing a fully MMR design that addressed all components of the MMR process. Johnson encouraged researchers to provide at least a brief indication of their philosophical beliefs, so that readers have a better understanding of the researcher’s perspectives.

Multiple Stakeholder Legitimation

Johnson discussed the importance of not only understanding the typical needs, but also addressing the needs and interests of those with minimal power. Considering multiple stakeholder legitimation, according to Johnson, “forces you to realize there will be different perspectives” and provides an opportunity to proactively address those perspectives.

Multiple stakeholder legitimation was not commonly addressed among our sample. Cooper explained that from a political standpoint he wanted to demonstrate the value of the quantitative and qualitative strands. Similarly, Onwuegbuzie et al. (2014) focused on enhancing the rigor of quantitative and qualitative techniques to address multiple stakeholder legitimation. Schoonenboom described the value of multiple stakeholder legitimation in informing others who can benefit from a study. She argued that identifying subgroups and then “exploring the groups where something is happening” might form the basis for an intervention, thereby extending the impact of the study beyond the participants.

Sequential Legitimation

Although the name sequential legitimation remained consistent across time, the definition heavily shifted. The original definition was the degree to which researchers have minimized potential problems in a sequential design that could arise from reversing the sequence of the quantitative and qualitative strands to generate meta-inferences. Johnson described a broader view of sequential legitimation by encouraging researchers to “think about how following strands in an MMR study build on earlier strands.”Johnson clarified that although sequential legitimation primarily relates to sequential designs, “if there are sequential components or procedures happening in a concurrent design it certainly applies to that, too.” Therefore, researchers should first identify whether they have implemented a sequential MMR design and if not, whether sequential components in a concurrent MMR design were incorporated to determine the relevance of this legitimation type. Onwuegbuzie, Frels et al. (2011) conducted a concurrent mixed methods design and collected data from two participant groups concurrently. Conducting simultaneous data collection helped to minimize validity issues.

Sequential legitimation was one of the least addressed legitimation types within this sample, particularly because it was not relevant to many of our participants’ studies as described in the original typology. Cooper stated that since he did not implement a sequential design, he did not address sequential legitimation in his study. Although Schoonenboom’s MMIGA model is a sequential methodology, she explained that sequential legitimation was not relevant because the quantitative and qualitative strand each had different roles and reversing the sequence would be illogical. However, Schoonenboom suggested that conducting a study in a reversed way might illuminate components that originally were not considered.

Integration Legitimation

Johnson and Christensen (2017) added integration legitimation to encourage researchers to assess the quality of integration in a mixed methods study with the aim of developing more credible meta-inferences. Creating a standalone type underscores the importance of addressing integration. Johnson recommends that researchers explain how they integrated at varying stages of their study including data collection, analysis, and conclusions. If the meta-inferences are justified by integrated conclusions, then integration legitimation has been achieved.

Pragmatic Legitimation

Johnson described the need for adding pragmatic legitimation to address the “so what?” question at the end of a study. By answering “Was the research purpose met?” and “Was the research problem solved?” (Johnson & Christensen, 2017, p. 308), researchers can add to readers’ understanding. This, Johnson underscores, “whether the mixed methods research has some pragmatic utility or usefulness.” Since this legitimation type was added in 2017, none of our participants directly addressed this type.

Summary

Overall, there were similarities and differences among researchers’ interpretations and applications of the original legitimation typology. Participants shared similar views on multiple validities legitimation, weakness legitimation, emic-etic legitimation, conversion legitimation, and sequential legitimation. They also noted challenges in assessing some legitimation types. For example, one participant described the need for clear guidelines on determining how much data is necessary to represent emic and etic perspectives. Some researchers stated the need for additional analyses for data conversion. When discussing sequential legitimation, most participants believed that this applied to a sequential mixed methods design; however, Johnson clarified that it also relates to the sequencing of events in a convergent design.

There were notable differences on some interpretations such as sample integration legitimation, commensurability legitimation, multiple stakeholder legitimation, and paradigmatic legitimation. Although most participants interpreted sample integration legitimation as consistent with Johnson’s definition, one participant believed a true positivistic (e.g., random) sample was needed. Differences arose with commensurability approximation legitimation with one participant interpreting it as accounting for different worldviews, like Johnson’s description; however, some referred to it as one strand being complementary to the less dominant strand. Differing interpretations also arose with multiple stakeholder legitimation, with one participant describing it as those who would benefit from the study aside from participants, whereas another participant interpreted it as the value of each strand from a political viewpoint. Finally, interpretations of paradigmatic/philosophical legitimation differed with one participant stating that the quantitative and qualitative strands should be equal value with respect to inquiry and another participant related this legitimation to a mix of worldviews. Although some researchers shared similar interpretations of some legitimation types, other types precipitated varied interpretations.

Value of Divergent Findings

Divergent findings arise when a researcher discovers contradictory support between the quantitative results and qualitative findings through the process of mixing (Pluye et al., 2009). Although there were initially no specific questions addressing divergent findings in the interview protocols, this topic emerged during the first interview and was subsequently added to our protocol with the remaining participants.

Bustamante found that qualitative interviews did not corroborate results from participant surveys. While Bustamante described making sense of this divergence as a challenge, she also mentioned several advantages to what she termed “discordant data.” Through the process of exploring the divergent findings that emerged from her data, she realized problems with the wording of items on the questionnaire. Hence, one benefit of divergent findings is that it provides researchers the opportunity to assess quality more carefully at different stages of the research process. Dismuke echoed this advantage but added that while there were no divergent findings in their study, it was important to follow-up with participants to give them the opportunity to share their perspectives and any additional information to supplement the researcher’s observations. Schoonenboom identified another advantage of divergent findings as strengthening the rationale for conducting a mixed methods study. Lastly, Ivankova suggested that studies addressing divergent findings encourage researchers to pursue higher standards of quality by assessing validity at all levels, thereby benefitting the MMR field. Ivankova noted that although researchers seem reticent to report divergent findings, “we are not looking at it [divergent findings] as a weakness, but as something that is probably the advantage of mixed methods.” Ivankova recommended researchers reassess the mixed methods study at its most foundational level—the validity of the individual quantitative and qualitative strands. She also encourages researchers to address two central questions when divergent findings arise: (a) Are the findings divergent because of a quality issue? or (b) Are the findings divergent because they are “revealing something new and unique” pertinent to the mixed methods design?

Summary

Most participants expressed advantages to divergent findings in MMR. For example, one participant was able to identify potential issues with their survey, while another expressed the advantage of delving deeper into participants’ perspectives to determine whether findings corroborated with researchers’ interpretations. In MMR, divergent findings can expand on current findings or shed light on potential threats to validity.

Strategies for Applying the Legitimation Typology

Several participants offered strategies to future researchers using the legitimation typology. Johnson encouraged researchers to consider QUAN validity, QUAL trustworthiness, and mixed methods legitimation throughout planning, designing, and conducting a study. Thoughtful application of the legitimation typology throughout the process can help researchers make stronger design choices that address potential threats to validity. Cooper suggested that researchers should confirm which legitimation types are relevant at the conclusion of a study, given the often-emerging nature of mixed methods designs in practice.

As an exercise to learn about the application of the legitimation typology, Cooper advised that while reading MMR studies, scholars should write memos explicitly identifying the legitimation types they encounter and thoughts on how examples might apply to the reader’s own research. Cooper advocated for developing example studies that address specific legitimation types as another way for scholars to learn more about the legitimation typology in practice. Finally, Cooper recommended scholars and researchers attempt to address all the legitimation types in their study. Clearly not all legitimation types are relevant to all studies, but before dismissing a particular type as irrelevant, Cooper encouraged researchers to imagine how they might adapt their study to make that type relevant. This exercise not only helps reduce confirmation bias but also expands potential design decisions that might not have been previously considered.

Swedler also offered advice to researchers implementing the legitimation typology. He encouraged researchers to carefully consider adding a second method to a study. He noted, “if you’re going to go through with it, what benefit will that add to your total results?”Swedler acknowledged the importance of having the appropriate training and resources before undertaking an MMR study. He encouraged researchers to “make sure that you are capable of conducting both methods or that you have a steady team capable of conducting both methods and you are ready to produce that added value that you expect to find by adding the second method.”

Ivankova identified the importance of considering multiple MMR quality frameworks for scholars who want to learn more about applying the legitimation typology. Making a purposeful choice to use one, or a combination of frameworks, allows for a stronger alignment with the research purpose and will contribute to the growing conversations about quality in MMR. Ivankova suggested scholars describe the quantitative and qualitative quality components separately, in the context of reflecting on their worldview, to examine how these criteria are applied in the scholars’ field. Particularly, encouraging students to reflect on their research paradigms allows them to feel more comfortable with quality perspectives that exist in their fields. Finally, Ivankova advocated researchers address the quality of a mixed methods study within grant proposals, whether it is specifically called for or not. Detailing a quality framework, such as the legitimation typology, demonstrates a more thorough examination of potential validity threats before conducting the study.

Summary

Several suggestions were provided on the applicability of the legitimation typology including consideration of legitimation types most pertinent to a study, the value of a second methodology, and providing a sound rationale for using one or several quality framework(s). One participant provided recommendations on ways researchers could apply the legitimation typology to better understand each type. This exercise can be particularly beneficial to students learning about quality frameworks in MMR.

Discussion

Contribution to Mixed Methods Research Literature

As the first empirical study to examine the use of the legitimation typology, we contribute an understanding of how legitimation is being interpreted and applied in practice. In the 15 years, since it was proposed by Onwuegbuzie and Johnson (2006), the application of the legitimation typology was only discussed in 49 published empirical MMR studies. Although it would be difficult to calculate how many empirical MMR studies were published during this time frame, these findings suggest that the use of the legitimation typology was extremely limited. Alternatively, many of the issues identified in the legitimation typology may be addressed in practice but excluded from publications. This suggests that despite its frequent description in MMR texts, researchers may either be unfamiliar with the legitimation typology or fail to prioritize its inclusion in limited publication space. We offer the following suggestions based on the study’s findings to encourage the application and discussion of the legitimation typology in empirical MMR studies.

Develop Guidance on Procedures When Converting Data

There is a dearth of practical guidance on ways to quantitize and qualitize data in a rigorous way. Several participants shared their uncertainty in how to convert data and identify appropriate analysis techniques. One suggestion for researchers using principal component analysis or factor analysis is to use the factors or components extracted from these models as a validation tool once all themes have been analyzed. We believe this would provide additional evidence of trustworthiness of the original qualitative findings. For example, a researcher could compare the factors or components with the qualitative findings to determine the level of convergence and divergence if examining similar research questions. Quantitative factors or components absent in the qualitative themes provide an opportunity for additional exploration of the data or additional rounds of data collection. Although this would not be considered data conversion, it could enhance the validity of the findings if these analyses were incorporated in a study. The mixed analysis approach of factor analysis of text data can be conducted using the combination of QDA Miner and WORDSTAT (Provalis Research, 2020). We also encourage researchers to consult The Routledge Reviewer’s Guide to Mixed Methods Analysis (Onwuegbuzie & Johnson, 2021) for the latest in innovative analytical methods for MMR, including data conversion approaches for quantitative, qualitative, and mixed methods analyses, and software for conducting mixed methods analyses. We recommend that MMR scholars continue to develop and promote best practices for analyzing converted data.

Determine Procedures to Assess Appropriate Balance for Emic-Etic (Inside-Outside) Legitimation

Some participants discussed the difficulty in determining the appropriate balance between the insider and the outsider perspectives when addressing emic-etic legitimation and how to resolve discrepancies between these two perspectives. Yin (2010) posited that discrepancies between the emic and etic perspectives exist due to researcher reflexivity, research design, and ultimately how researchers reported their study. Attending to reflexivity and careful design are necessary to achieve emic-etic legitimation. This is especially critical if a researcher has a relationship to a group they are studying or shares similarities with them, as unexpected differences may arise (Olive, 2014). Johnson and Christensen (2017) expanded this discussion by noting several strategies for defensible qualitative/emic findings including member checks. To improve the quality of the researcher’s etic perspective, the researcher can use multiple investigators to analyze and interpret data, and continually search for and attempt to dismiss alternative explanations of findings. Emic-etic legitimation requires the production of both viewpoints and merging them into meta-viewpoints. We encourage additional exploration of this topic by MMR scholars to guide future researchers in attending to this legitimation type.

Encourage Adoption of the Refined Sequential Legitimation Definition

As pointed out by Schoonenboom, reversing the sequence of the quantitative and qualitative strands in sequential designs is often illogical. By definition, the individual strands of the basic sequential MMR designs (e.g., explanatory and exploratory) depend upon each other such that the initial strand must be completed in order for the second strand to commence. The original conceptualization of sequential legitimation as addressing issues if the sequencing of the strands were reversed was counterintuitive for researchers to implement in these basic designs. That confusion might explain why so few researchers discussed sequential legitimation. However, the revised definition of sequential legitimation (Johnson & Christensen, 2020) eliminates this confusion by focusing on the appropriateness of building from the initial strand to the follow-up strand instead of reversing their order. The revised definition also addresses cases where “you might not want a later stage to be affected by a prior stage because the ordering was arbitrary” (2020, p. 290). We support expanding this part of the definition to include potential validity threats of priming effects of the order of data collection (not just stages or phases of research), thereby extending sequential legitimation to encompass a broader array of sequential issues. We encourage researchers to become more familiar with the updated legitimation typology, specifically for sequential legitimation, as it is more easily applicable in practice.

Create Divergent Findings Legitimation

Many of our participants noted the challenge of addressing divergent findings when conducting MMR. Some researchers may overlook or completely dismiss their findings when faced with contradictory results (Patton, 2002). Nevertheless, one true value of MMR shines in those moments when researchers wrestle with the conundrum caused by divergent findings and a deeper understanding of the phenomenon emerges. Given the saliency of divergent findings across participant interviews and its implications in MMR, we recommend the addition of divergent findings legitimation to the legitimation typology. Adding divergent findings legitimation will further reinforce researchers to investigate the validity, trustworthiness, and quality at all levels of their study—at the macrolevel (e.g., quantitative, qualitative, and mixed methods) and the microlevel (e.g., design elements, measures, data collection, and data analysis) when faced with divergent findings.

Creamer and Edwards (2018) reimagined the categories of responses to divergent findings proposed by Pluye et al. (2009) as a decision tree that arranges the categories as a series of inter-linked steps. They described the process as starting with bracketing, which means that the researcher determines if the quantitative and qualitative strands are addressing different aspects of the phenomenon, and therefore, the divergent findings are not problematic. If this is not the case, researchers then consider exclusion or weighting, in which they review the rigor of the individual strands and increase the relative priority of the strand with a stronger case for validity. When the divergent findings cannot be attributed to weak validity evidence of one of the strands, researchers move to reanalysis to reconcile, which involves conducting new analyses or re-engaging with the literature. The final step in this process is to initiate new data collection to resolve or confirm the divergence. The processes proposed in this article and by Creamer and Edwards could serve as a framework to address divergent findings legitimation.

Clarify Context for Using Each Legitimation Type

Throughout our methodological review and interviews, it became clear that MMR scholars and researchers had differing interpretations on the relevancy of particular legitimation types. Thus, we identified some “missed opportunities” where researchers should have addressed a broader array of legitimation. To avoid this oversight in the future, we recommend that the context for using each legitimation type be clarified so that researchers can more readily identify which types should be addressed in their work. Based on our understanding of the legitimation typology across its three versions, its use in published MMR studies, and data from MMR scholars and researchers, we propose initial contextual guidelines (Table 5). We encourage others to share their work exploring and expanding upon these contexts.

Table 5.

Context for Using Each Legitimation Type.

Legitimation type	Context for using legitimation type
Multiple validities	All MMR studies
Sample integration	Most MMR studies
Weakness minimization	Most MMR studies
Emic-etic	Most MMR studies
Commensurability approximation	Most MMR studies
Conversion	Most MMR studies that quantitize or qualitize data for cross-over analysis
Multiple stakeholder	Most MMR studies, especially those with a social justice orientation
Paradigmatic/Philosophical	Most MMR studies
Sequential	Most MMR studies that are sequential or involve sequential components
Integration*	All MMR studies
Pragmatic*	Most MMR studies, especially those with an applied or action research orientation

Conclusion

Summary of Major Findings

The purpose of this intrinsic, exploratory case study was to explore researchers’ perspectives on quality in MMR specifically focusing on the legitimation typology. We conducted an MMR-SMR and interviews with seven key MMR scholars and researchers. We identified 49 published empirical MMR studies that discussed the use of the legitimation typology in the 15 years since the typology was proposed. To explore and understand researchers’ interpretations and applications of the legitimation typologies, each participant provided their interpretations and use of specific legitimation types of their mixed methods study. It was evident that researchers have unique interpretations on some legitimation types that influenced whether they were addressed in the study. Many researchers expressed uncertainty on whether their interpretations were consistent with the typology. Four themes were identified from the interviews: (a) comprehensive approach to assessing quality, (b) researchers’ interpretation of legitimation types, (c) value of divergent findings, and (d) strategies for applying the legitimation typology. Many participants agreed that validity should be assessed throughout the entire research process, especially during the planning stage, and each strand as well as the full mixed methods study. Based on these findings, we recommend that the field of MMR provide guidance on how to convert data in MMR studies and how to assess the balance between insider and outsider perspectives. We suggest MMR scholars, editors, and publishers encourage the use of the most current legitimation typology (Johnson & Christensen, 2020), particularly the updated definition of sequential legitimation. Further, we propose the addition of divergent findings legitimation and contextual information to guide the use of each legitimation type more clearly.

Limitations and Future Directions

One limitation of this study was the inability to access and screen 687 potential articles from the Google Scholar search. Had these articles been accessible, we would have been able to determine if articles from this group met inclusion criteria and added to the overall sample. In addition, while we found that increased description lengths and a table were associated with a higher number of legitimation types addressed, this finding is limited by a journal’s word limit. Future studies should investigate this by comparing across journals with varying word limits.

Another limitation of our study is the lag time between conduct of research and its publication. At the time of our interviews, published empirical studies had only cited the original 2006 legitimation typology. Our subsequent literature search identified only two studies (Liu et al., 2019; Younas et al., 2020) that cited the updated 2017 version. Hence, our study predominantly represents how the original typology was implemented in practice. We encourage the continued monitoring on the use of the legitimation typology to note how ongoing discussions and application influence its adaptations. Finally, there exists challenges when reproducing results across different queries in Google Scholar and it typically excludes many common features that are required by several databases (Gusenbauer & Haddaway, 2020). Nevertheless, Google Scholar was mostly used as a supplementary source in this study. As the use of MMR continues to increase, we anticipate discussions on quality in MMR will also increase. Thus, by elucidating researchers’ implementation of the legitimation typology to assess the quality of an MMR study, we contribute to this continuing methodological dialogue.

Supplemental Material

sj-docx-1-mmr-10.1177_15586898221093872 – Supplemental material for Advancing Quality Standards in Mixed Methods Research: Extending the Legitimation Typology

Supplemental material, sj-docx-1-mmr-10.1177_15586898221093872 for Advancing Quality Standards in Mixed Methods Research: Extending the Legitimation Typology by Analay Perez, Michelle C. Howell Smith, Wayne A. Babchuk and Louise I. Lynch-O’Brien in Journal of Mixed Methods Research

Footnotes

Acknowledgements

The authors would like to thank the participants who took part in this study, without them, this study would not have come to fruition. In addition, the authors would also like to thank all reviewers and editors who provided instrumental feedback throughout all revisions of this manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Analay Perez

Michelle C. Howell Smith

Louise I. Lynch-O’Brien

Supplemental Material

Supplemental material for this article is available online.

References

Bustamante

(2019). TPACK and teachers of Spanish: Development of a theory-based joint display in a mixed methods research case study. Journal of Mixed Methods Research, 13(2), 163–178. https://doi.org/10.1177/1558689817712119

Collins

K .M. T.

Onwuegbuzie

A. J.

Johnson

R. B.

(2012). Securing a place at the table: A review and extension of legitimation criteria for the conduct of mixed research. American Behavioral Scientist, 56(6), 849–865. https://doi.org/10.1177/0002764211433799

Cooper

J. N.

Hall

(2016). Understanding black male student athletes’ experiences at a historically black college/university: A mixed methods approach. Journal of Mixed Methods Research, 10(1), 46–63. https://doi.org/10.1177/1558689814558451

Creamer

E. G.

Edwards

C. E.

(2018). Paradigms in play: Using case studies to explore the value-added of divergent findings in mixed methods research. International Journal of Multiple Research Approaches, 10(1), 30–40. https://doi.org/10.29034/ijmra.v10n1a2

Fàbregues

Molina-Azorín

J. F.

(2017). Addressing quality in mixed methods research: A review and recommendations for a future agenda. Quality & Quantity, 51(6), 2847–2863. https://doi.org/10.1007/s11135-016-0449-4

Fetters

M. D.

Freshwater

(2015). The 1+1= 3 integration challenge. Journal of Mixed Methods Research, 9(2), 115–117. https://doi.org/10.1177/1558689815581222

Goodwin

Polek

Goodwin

(2013). Perceived changes in health and interactions with “the paracetamol force”: A multimethod study. Journal of Mixed Methods Research, 7(2), 152–172. https://doi.org/10.1177/1558689812463782

Greene

J. C.

(2006). Toward a methodology of mixed methods social inquiry. Research in the Schools, 13(1), 93–98.

Guest

Bunce

Johnson

(2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), 59–82.

10.

Gusenbauer

Haddaway

N. R.

(2020). Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods, 11(2), 181–217.

11.

Haddaway

N. R.

Collins

A. M.

Coughlin

Kirk

(2015). The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PloS One, 10(9), Article e0138237.

12.

Howell Smith

M. C.

Shanahan Bazis

(2020). Conducting mixed methods research systematic methodological reviews: A review of practice and recommendations. Journal of Mixed Methods Research. https://doi.org/10.1177/1558689820967626

13.

Howell Smith

M. C.

Babchuk

W. A.

Stevens

Garrett

A. L.

Wang

S. C.

Guetterman

(2019) Modeling the use of mixed methods-grounded theory: Developing scales for a new measurement model. Journal of Mixed Methods Research. https://doi.org/10.1177/1558689819872599

14.

Janesick

V. J.

(2016). Stretching exercises for qualitative researchers (4th ed.). Sage.

15.

Johnson

R. B.

Christensen

L. B.

(2017). Educational research: Quantitative, qualitative, and mixed approaches (6th ed.). Sage.

16.

Johnson

R. B.

Christensen

L. B.

(2020). Educational research: Quantitative, qualitative, and mixed approaches (7th ed.). Sage.

17.

Kawamura

Ivankova

N. V.

Kohler

C. L.

Perumean-Chaney

(2009). Utilizing mixed methods to assess parasocial interaction of an entertainment–education program audience. International Journal of Multiple Research Approaches, 3(1), 88–104. https://doi.org/10.5172/mra.455.2.1.88

18.

Liu

X. L.

Willis

C. J. J.

Fulbrook

Shi

Johnson

(2019). Preparing Chinese patients with comorbid heart disease and diabetes for home management: A mixed methods study. BMJ Open, 9(9), Article e029816. https://doi.org/10.1136/bmjopen-2019-029816

19.

Martin

S. D.

Dismuke

(2018). Investigating differences in teacher practices through a complexity theory lens: The influence of teacher education. Journal of Teacher Education, 69(1), 22–39. https://doi.org/10.1177/0022487117702573

20.

Merriam

S. B.

(1988). Case study research in education: A qualitative approach. Jossey-Bass.

21.

Olive

J. L.

(2014). Reflecting on the tensions between emic and etic perspectives in life history research: Lessons learned. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 15(2), Art. 6. http://nbn-resolving.de/urn:nbn:de:0114-fqs140268

22.

Onwuegbuzie

A. J.

(2012). Introduction: Putting the MIXED back into quantitative and qualitative research in educational research and beyond: Moving toward the radical middle. International Journal of Multiple Research Approaches, 6(3), 192–219.

23.

Onwuegbuzie

A. J.

Frels

R. K.

Leech

N. L.

Collins

K. M.

(2011a). A mixed research study of pedagogical approaches and student learning in doctoral-level mixed research courses. International Journal of Multiple Research Approaches, 5(2), 169–199. https://doi.org/10.5172/mra.2011.5.2.169

24.

Onwuegbuzie

A. J.

Johnson

R. B.

Collins

K. M.

(2011b). Assessing legitimation in mixed research: A new framework. Quality & Quantity, 45(6), 1253–1271. https://doi.org/10.1007/S11135-009-9289-9

25.

Onwuegbuzie

A. J.

Johnson

R. B.

(2006). The validity issue in mixed research. Research in the Schools, 13, 48–63.

26.

Onwuegbuzie

A. J.

Johnson

R. B.

(Eds.). (2021). The routledge reviewer’s guide to mixed methods analysis. Routledge.

27.

Onwuegbuzie

A. J.

Rosli

Ingram

J. M.

Frels

R. K.

(2014). A critical dialectical pluralistic examination of the lived experience of select women doctoral students. Qualitative Report, 19(3), 1–35. https://nsuworks.nova.edu/tqr/vol19/iss3/1

28.

Onwuegbuzie

A. J.

Witcher

A. E.

Collins

K. M.

Filer

J. D.

Wiedmaier

C. D.

Moore

C. W.

(2007). Students’ perceptions of characteristics of effective college teachers: A validity study of a teaching evaluation form using a mixed-methods analysis. American Educational Research Journal, 44(1), 113–160. https://doi.org/10.3102/0002831206298169

29.

Patton

M. Q.

(2002). Qualitative research and evaluation methods. Sage.

30.

Plano Clark

V. L.

Ivankova

N. V.

(2016). Mixed methods research: A guide to the field. Sage

31.

Pluye

Grad

R. M.

Levine

Nicolau

(2009). Understanding divergence of quantitative and qualitative data (or results) in mixed methods studies. International Journal of Multiple Research Approaches, 3(1), 58–72. https://doi.org/10.5172/mra.455.3.1.58

32.

Provalis Research. (2020). QDA Miner: Qualitative and mixed-method software (version 5.0) [software]. http://provalisresearch.com/downloads/trial-versions/

33.

Saldaña

(2021). The coding manual for qualitative researchers (4th ed.). Sage.

34.

Schoonenboom

(2016). The multilevel mixed intact group analysis: A mixed method to seek, detect, describe, and explain differences among intact groups. Journal of Mixed Methods Research, 10(2), 129–146. https://doi.org/10.1177/1558689814536283

35.

Smith

C. R.

Gillespie

G. L.

(2016). Do adolescent employees perceive the risks of workplace violence? A mixed methods study. Journal of Nursing Education and Practice, 6(7), 116–124. https://doi.org/10.5430/jnep.v6n7p116. Software. https://www.maxqda.com

36.

Stake

R. E.

(1995). The art of case study research. Sage.

37.

Swedler

D. I.

Pollack

K. M.

Gielen

A. C.

(2015). Understanding commercial truck drivers’ decision-making process concerning distracted driving. Accident Analysis & Prevention, 78, 20–28. https://doi.org/10.1016/j.aap.2015.02.004

38.

Torres-Alzate

H. M.

Wilson

L. L.

Harper

D. C.

Ivankova

N. V.

Heaton

Shirey

M. R.

(2020). Essential global health competencies for baccalaureate nursing students in the United States: A mixed methods Delphi study. Journal of Advanced Nursing, 76(2), 725–740. https://doi.org/10.1111/jan.14030

39.

VERBI Software. (2017). MAXQDA 2018 [computer software]. Berlin, Germany: VERBI.

40.

Yin

R. K.

(2010). Qualitative research from start to finish. The Guilford Press.

41.

Younas

Rasheed

S. P.

Zeb

(2020) Using legitimation criteria to establish rigour in sequential mixed-methods research. Nurse Researcher, 28(3), 44–51. https://doi.org/10.7748/nr.2020.e1727

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB