Abstract
Mixed methods research (MMR) methodologists research the use of MMR by conducting systematic methodological reviews (SMR). Unfortunately, there is little guidance on how to conduct these types of reviews. This MMR-SMR of MMR-SMRs identifies a comprehensive pool of published MMR-SMRs (n = 80), isolating best practices that will guide future reviews thereby increasing their quality and utility to the field of MMR. We summarize our quantitative codes descriptively and integrate the qualitative themes alongside the relevant quantitative results. We contribute to the field mixed methods research by identifying potential topics for future MMR-SMR, summarizing current practices in writing MMR-SMRs, and providing recommendations to consider when publishing future MMR-SMR.
Understanding how mixed methods research (MMR) is being used by different disciplines or to address specific research topics is of great interest to MMR scholars. MMR scholars study the use of MMR by conducting literature reviews of empirical MMR studies (Molina-Azorín & Fetters, 2016). These reviews typically identify a pool of relevant empirical studies then analyze them for key methodological issues. Such reviews are valuable because they not only identify trends in the use of MMR but also provide important information to inform and improve future MMR studies (e.g., Coyle et al., 2018; DeJonckheere et al., 2018; Guetterman & Fetters, 2018). Thus, these reviews elevate the legitimacy of MMR studies, particularly in fields that may be more entrenched in mono-method approaches.
While there have been a growing number of these types of reviews, locating them can be a challenge due to the lack of standardized naming conventions and the diversity of journals in which they appear. Therefore, the utility of these reviews is limited if their findings are not widely known and it is difficult to separate reviewed disciplines and topics from those that would benefit from a review. Furthermore, there is little guidance to MMR scholars for conducting these types of literature reviews; hence, the scope of analysis and level of detail provided in these reviews are quite variable.
The purpose of this study is to create a census of existing MMR systematic methodological reviews and identify best practices that will guide future reviews in order to increase their visibility and value to the field of MMR. Thus, we conducted an MMR systematic methodological reviews of published MMR systematic methodological reviews. The following research questions guided our inquiry:
This study provides important insights for both MMR researchers and methodologists because it summarizes the current practices in writing reviews, identifies potential topics for reviews, and provides recommendations to consider when publishing reviews.
What Is a Mixed Methods Research Systematic Methodological Review?
MMR methodological reviews are not always easy to identify in the literature as there is not a standard naming convention. These reviews have been called a variety of terms including content analysis, mapping review, narrative review, scoping review, systematic literature review, and systematic review. Plano Clark and Ivankova (2016) identified three primary types of MMR methodological reviews: advocacy writings, disciplinary-based discussions, and systematic methodological reviews. The purpose of advocacy writing is increasing the stature of MMR within a field of study. We would argue that all MMR methodological reviews serve an advocacy role, and therefore the distinction of “advocacy” as a type of review has minimal utility. Disciplinary-based discussions provide specific advice on how to implement MMR within a field. Given that many methodological reviews provide tacit or explicit recommendations for how to use (or improve the use of) MMR within a discipline or field, categorizing reviews based on providing advice on how to conduct MMR studies is likewise of limited value. MMR systematic methodological reviews summarize trends in how MMR is used within a field or across fields using a pool of studies identified in a systematic way. Prevalence studies are a subset of systematic methodological reviews. In addition to the general intent of systematic methodological reviews, prevalence studies categorize a bounded group of empirical studies as quantitative, qualitative, or MMR in order to calculate the percentage of studies that are MMR (Molina-Azorín & Fetters, 2016). Noting the inconsistent naming conventions and overlapping definitions in the literature, we propose the following typology in Table 1 for categorizing methodologically focused reviews.
Types of MMR Methodological Reviews.
Note. MMR = mixed methods research; SMR = systematic methodological review.
Method
Data Collection Process
For this study, we were interested in how authors of MMR systematic methodological reviews (including prevalence studies) use examples of empirical MMR studies to educate their respective audiences about the use of MMR related to a topic, discipline, or issue. We began our data collection process by identifying inclusion criteria for our MMR-SMR. We strove for an inclusive approach that would help us identify articles across varied subjects and authors to gain a complete picture of both the development of and the current state of MMR methodological research. We limited our scope to articles published in peer-reviewed journals. We excluded book chapters, full books, theses, and dissertations from our review in order to have consistency in terms of scope and length.
To identify potential reviews, we used a variety of approaches, as depicted in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) diagram (Figure 1). We conducted a systematic literature search using EBSCO Host databases (Academic Search Premier, PsycINFO, and PsycARTICLES). These databases were selected based their comprehensive content of sociological, behavioral, and educational research literature. We searched abstracts using the terms “mixed method*” and either “review” or “prevalence.” This search yielded 2,913 records; 57 were retained. We also identified a pool of 68 articles using nonsystematic approaches. We reviewed the full text of these 125 publications, retaining 80 reviews in our final pool, 40 from the systematic search and 40 from nonsystematic sources.
PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) diagram of MMR-SMR identification process.
Data Analysis
We adapted the initial coding protocol from the grading rubric and coding protocol used for an MMR-SMR assignment in an introduction to MMR course taught by the first author. For the assignment, students identified MMR studies in their research area and coded them for quantitative and qualitative design features such as type of data collected, analysis procures, participant n and characteristics, as well as basic mixed methods design features such as timing and form of integration, rationale for using MMR, use of MMR literature, and the value-added contribution of the MMR study. To this protocol, we added in literature review elements such as search engines, key words, and date restrictions. To refine the initial coding protocol, we independently coded three randomly selected MMR-SMRs. We compared notes to establish initial agreement about the codes and how they would be applied, updated code definitions, and added new codes that emerged. The remaining 77 reviews were listed in alphabetical order and divided in an alternating fashion between us for coding so that each review was coded by one of us. We purposively selected reviews throughout the coding process that were particularly difficult to code (n = 18) for review by a second coder, as modeled by Powell et al. (2008). We discussed all recommended changes to the original coding and updated our code book based on these ongoing discussions to further clarify our coding process. Due to the evolving nature of the codebook throughout the initial coding process, all reviews were ultimately recoded by both of us to ensure consistency with the final codebook and agreement between the coders. The final coding protocol, full references for reviewed studies, and code book are available in the online supplementary material.
Using a Microsoft Excel spreadsheet as a coding table, we first coded each review for general characteristics such as publication year, journal, authors, general content area, type of review, and research questions. We then coded the procedures used in the review to identify studies, including databases used, search terms, finalizing the pool of studies analyzed, and the publication date range of empirical studies analyzed. For reviews that calculated a prevalence rate, we noted the proportion of MMR studies, recalculating as needed to include only empirical articles. Next, we focused on the analysis conducted in each review, such as interrater agreement procedures, MMR topics examined in the review, completeness of the coding results, and the use of MMR literature. Finally, we noted if the reviews discussed any limitations and whether the authors of the reviews made any recommendations for future MMR studies.
We also copied verbatim any text related to the following five topics in a separate excel tab for each topic: explicit research questions, rationale for selecting the publication date range, rationale for selecting search terms to identify MMR studies, limitations of both the studies included in their review and limitations of their review methodology; and explicit and implicit recommendations for future MMR empirical studies. We coded each topic to reflect the general content. We inserted additional columns in our spreadsheet (one for each code) to be able to sort reviews by codes, thus facilitating comparisons and contrasts across the pool of reviews. The initial codes were combined or revised in an iterative process until we had small handful of overarching themes that seemed to capture the essence of each topic. Text segments that provided vivid description of a topic were noted for use as examples. All codes and themes were reviewed and discussed throughout the analysis process.
At the conclusion of our data analysis, we summarized our codes descriptively and integrated the themes alongside the relevant results.
Results
Overview of MMR-SMRs
Publication Year
The 80 MMR-SMRs in our study were published over the course of 31 years as depicted in Figure 2. The first review (Greene et al., 1989) remained the only review for 14 years. The next review was published in 2004, and the number grew steadily each year, with notable peaks between 2010 and 2012.
Number of MMR-SMRs published by year.
Journals
The 80 MMR-SMRs in our pool were published in 58 different journals. In all but four cases, each journal had published only one review. Both Quality & Quantity and Qualitative Inquiry published two (3%) reviews each, while the International Journal of Multiple Research Approaches published 10 (13%) reviews and the Journal of Mixed Methods Research published 13 (16%).
Authors
Although the majority (n = 66, 83%) of the MMR-SMRs in our pool were written by multiple authors, 14 (18%) were written by a single author. Seventeen scholars contributed to more than one review, such as Onwuegbuzie (n = 8), Molina-Azorín (n = 7), and Plano Clark (n = 7). See the online supplementary material for a complete list of authors with contributions to multiple MMR-SMRs.
General Content Area
Whereas the reviews addressed a wide array of disciplines and topics, we coded each review into the following broad categories: business, education, health, social science, and methodological issues. Table 2 indicates the number and percentage in each category. The methodological issues that were examined in our pool of reviews included rationales, funding patterns, sampling designs, approaches to integration, longitudinal MMR designs, mixed methods grounded theory, mixed methods case studies, community-based participatory research, transformative MMR, art-MMR, inferences, collaboration patterns, addressing divergent findings, synthesis-level MMR, data transformation, quality, complexity and rigor, the use of MMR literature, and the added value of MMR.
General Content Area of Reviews.
Note. Total N = 80. Some reviews were in more than one category.
Type of Review
Authors of the reviews in our pool demonstrated a lack of agreement in how they identified their reviews. Overall, authors of 64 (80%) reviews indicated a name for the review or type of study they were conducting. Most commonly (n = 37, 46%), these articles were referred to as a review, such as systematic, methodological, literature, narrative, scoping, critical, or empirical reviews. Authors of 13 (16%) of the reviews framed them as an analysis including content, genre, interpretive, or documentary analysis. Authors also described their review as a study (n = 12, 15%) such as prevalence, qualitative, mixed methods, exploratory, research, or bibliometric studies. A review of the use of the transformational framework in mixed methods studies was named an advocacy writing by the authors (n = 1, 1%; Sweetman et al., 2010). Another methodological review was framed as a research synthesis (n = 1, 1%; Heyvaert et al., 2011). Authors of 16 (20%) of the reviews did not indicate the type of review they were conducting.
Research Questions
Approximately half (n = 41, 51%) of the reviews provided at least one research question. Our analysis of the stated research questions identified two primary themes: “How has MMR been used in “X” field or topic?” and “What is the prevalence rate of MMR in “X” journal(s) over “X” time period?” for prevalence rate studies. Reviews with research questions about the use of MMR often included subquestions related to identifying the MMR design based on an established typology. A small handful of these reviews asked specific subquestions about key decisions such as types of data collected, timing, priority, integration, and so on.
In addition to the two main themes of research questions, some reviews posed research questions about quality and rigor of the studies, while others focused on impact and contributions. More specific research questions were reported in several reviews, particularly those that focused on methodological issues. One review of how MMR studies address divergent findings asked, “What strategies are used by authors of articles explicitly framed with a purpose that fits the traditional definition of the initiation rationale to reconcile divergent findings?” (Creamer, 2018, p. 31). A review of the use of MMR in longitudinal designs asked, “What issues occur during the conduct of longitudinal mixed methods research?” (Plano Clark et al., 2015, p. 298). A review regarding the degree of collaboration in select mixed methods research journals included a qualitative research question, “What are select leading mixed researchers’ experiences conducting mixed research alone versus conducting mixed research in teams?” (Onwuegbuzie et al., 2018, p. 440) that led to the collection of interview data in addition to their methodological review.
Process Used to Identify Empirical Studies for MMR-SMRs
Authors of the reviews in our pool employed diverse strategies to identify relevant studies for the reviews they were conducting. Given the different nature of identifying empirical MMR studies, we first present the findings for MMR-SMRs that were not prevalence studies, followed by MMR prevalence studies (MMR-PSs).
Strategies to Identify Empirical Studies for MMR-SMRs (Nonprevalence)
Of the 49 nonprevalence MMR-SMRs, 35 (71%) identified studies through a traditional database search, 15 (31%) through searching specific journals, and 12 (24%) used other search approaches. Nine (18%) of the reviews used multiple approaches to identify potential studies. Other search approaches included scanning reference lists to identify potential studies (e.g., Boeije et al., 2013; Heyvaert et al., 2011; Sahin & Naylor, 2017), using studies the authors were already familiar with (Guetterman et al., 2019), selecting studies identified by previous reviews (e.g., Cameron & Molina-Azorín, 2011; Collins et al., 2006), and purposively selecting articles, reports, and articles (Greene et al., 1989).
Databases Used
A variety of databases were used to identify empirical studies for MMR-SMRs. Of the 35 reviews that reported using a database as part of their search, PsycINFO/PsycARTICLES was used most frequently, followed closely by PubMed and CINAHL. The number of databases used for each review ranged from one to 18, with a mean of 3.5. Table 3 provides the number of reviews that used each database if it was mentioned in more than one review. Twenty-two additional databases were used in one review each.
Databases Used as Part of the Search Process.
Note. n = 35.
Search Terms
Of the 49 nonprevalence MMR-SMRs, 33 (67%) provided the methodological search terms used to identify MMR empirical studies. We categorized terms in four broad categories: mixed method, multimethod, quantitative + qualitative, and methodological. We noted the specific variations within those categories. Variations of the term “mixed methods” appeared in 30 (91%) of the reviews. The most commonly used variations were “mixed methods,”“mixed method,” and “mixed method*.” Some form of multimethod (“multimethod,”“multi-method,” and “multiple methods”) was used in 13 (39%) reviews. Combinations of quantitative and qualitative terms were used in 18 (55%) of the reviews, and other methodological terms, such as triangulation, were used in 11 (33%) of the reviews.
We analyzed the stated justifications for methodological terms selected. One theme was related to pool size; limiting the number of identified empirical studies so the review would not be unwieldy or selecting terms to be as comprehensive as possible. Another theme was using search terms from previous reviews or recommendations from prominent MMR scholars. The final theme was selecting the specific MMR search terms to limit the review to studies that were explicitly conceptualized as mixed methods.
Finalizing the Pool of Studies Analyzed
The number of empirical MMR studies initially identified by the searches in the 49 nonprevalence MMR-SMRs ranged from five to 4,028 (M = 593; SD = 997). However, not all these studies were actually relevant to the review, either based on the content of the review or the methodology. Strategies to verify inclusion criteria and finalize the pool of empirical studies included examining only titles and keywords (n = 2, 4%); examining titles, keywords, and abstracts (n = 6, 12%); reviewing the method section or full article text (n = 29, 59%); or some other process (n = 2, 4%). Ten (20%) reviews only reported the final number of their pool; it is unknown if they conducted steps to verify the inclusion criteria or included all articles in their initial search. The importance of verifying the inclusion criteria was noted by Sweetman et al. (2010) who restricted their review to studies where they could “. . . locate the place in the article in which the two strands were integrated” (p. 443) in order to ensure the quality of the empirical studies in their review. Authors of 16 (33%) reviews provided a PRISMA diagram or other figures to illustrate their search process and detail the strategies they used to refine their pool of studies. The final number of empirical MMR studies analyzed in the reviews ranged from five to 232 (M = 48; SD = 51).
Strategies to Identify Empirical Studies for MMR-PSs
MMR-PSs generally identify empirical MMR studies by selecting a specific journal or small group of journals and then categorize all empirical studies as quantitative, qualitative, or mixed methods. Of the 31 MMR-PSs in our pool, the majority (n = 26, 84%) identified studies by focusing on a journal or set of journals. Three studies (10%) searched databases of funding agencies. One review searched for MMR-PSs and then synthesized the prevalence rates based on what was reported in the six reviews they identified in the field of business and management (Cameron & Molina-Azorín, 2011). Another review searched first for any study related to volunteering research and then categorized all studies identified in that content area to determine a prevalence rate (Paull & Girardi, 2017). The final number of empirical MMR prevalence studies analyzed in the reviews ranged from five to 654 (M = 101; SD = 131).
The MMR-PSs reported prevalence rates ranging from less than 1% to 34%, with an average prevalence rate of 14%. See Table 4 for the prevalence rates of MMR studies in broad fields of study.
MMR Prevalence Rates for Major Fields of Study.
Publication Date Range of Studies Analyzed
Three of the reviews did not provide enough information to determine the date range of the studies analyzed in their review. The remaining 77 reviews included studies published from 1950 through 2017. The span of the publication dates within each review ranged from 1 to 62 years (M = 11 years, SD = 10). We conducted a thematic analysis of the reasons provided for restricting the date range (n = 62, 78%). A predominant rationale was the publication of major literature in MMR such as the Handbook of Mixed Methods in Social & Behavioral Research (Tashakkori & Teddlie, 2003), Journal of Mixed Methods Research, or other seminal MMR publications. Another major theme was focusing on the current use of MMR. Other reasons for restricting the date rage were to ensure that the content was relevant, that there was either an adequate or manageable sample size, or because the authors were following an approach used in other methodological reviews. Of the 18 (23%) reviews that did not intentionally set the date range, some authors used other methods to bound their search process such as going back in time until they failed to identify any MMR studies (Plano Clark, 2010) or using studies identified in existing reviews (Cameron & Molina-Azorín, 2011; O’Cathain et al., 2008).
Analysis Conducted in MMR-SMRs
Interrater Agreement Procedures
Noting that the majority (n = 64, 80%) of the reviews in our study had more than one author, we coded for any discussion of interrater agreement procedures. Only slightly more than half (n = 37, 58%) of the multiauthor reviews addressed interrater agreement. Interestingly, two reviews with single authors (Fidel, 2008; Molina-Azorín, 2012) described interrater agreement procedures with a nonauthor colleague, hence, the percentages reported below are based on 39 reviews. The most common approach (n = 24, 62%) was a general description of research team meetings to discuss difficult studies to code or review codes assigned by multiple coders. Generally, this process resulted in consensus about how to resolve any discrepant codes (e.g., Bishop & Holmes, 2013; Hurmerinta-Peltomäki & Nummela, 2006). There were 10 (26%) reviews that reported an interrater agreement percentage (e.g., Powell et al., 2008; van der Roest et al., 2015) and five (13%) calculated a kappa statistic (e.g., Alise & Teddlie, 2010; Collins et al., 2006). The focus of the interrater agreement was generally on the coding procedures after the review pool had been identified (n = 32, 82%). However, three (8%) of the reviews only discussed interrater agreement in terms of the screening process for identifying studies to include in their review while four (10%) reviews commented on interreliability agreement for both screening and coding.
MMR Topics Examined in Reviews
There were several MMR topics that were examined by the 80 reviews in our pool. Table 5 includes MMR topics that were analyzed in at least 10% of the reviews. Issues such as quality indicators and citation impact were analyzed in less than 10% of the reviews. Two of the reviews did not describe any coding approach or present any coding results. Interestingly, authors of one MMR-PS did not actually calculate a prevalence rate; instead, they calculated the degree of collaboration among researchers in purposively selected quantitative, qualitative, and mixed methods journals and compared the collaboration rates across the three different methodological traditions (Onwuegbuzie et al., 2018).
MMR Elements Coded in MMR-SMRs.
Note. n = 80. MMR = mixed methods research; SMR = systematic methodological review.
Completeness of Coding Results
Unlike a descriptive table that summarizes data and their central tendencies, a code table provides the raw codes for each of the reviewed studies, thus providing direct access to the data so that other scholars may replicate or extend the review or further explore issues of interest. Half (n = 40, 50%) of the reviews in our study included complete coding results via a code table; the remaining reviews only summarized their coding results. While most MMR-PSs calculate the percentage of MMR studies based on the total of empirical studies (quantitative, qualitative, and MMR), some also include editorials, book reviews, and other publications in their calculations. A prevalence rate table clearly indicates the number of studies included in each category in order to facilitate direct comparison across other MMR-PSs. Nearly all MMR-PSs in our review (n = 27, 87%) included a complete prevalence rate table; the remaining prevalence studies did not provide sufficient details in their published article to determine if their prevalence rate was based only on empirical studies or if it included other types of publications.
Use of MMR Literature
One of the criteria for reviewing methodological articles for the Journal of Mixed Methods Research is the “adequacy of the literature” (n.d.). While we did not engage in a subjective evaluation of the MMR literature referenced in our pool of reviews, we did note the MMR scholars and MMR textbooks that were cited. The most cited scholars (found in 70% or more of the reviews) included Creswell, Greene, Johnson, Onwuegbuzie, Plano Clark, Tashakkori, and Teddlie. See the online supplementary material for the full list of the 26 most cited MMR scholars cited in our pool of reviews. Table 6 includes the MMR textbooks cited in more than 10% of our pool of reviews.
Mixed Methods Textbooks Cited in MMR-SMRs.
Recommendations Made in MMR-SMRs
Of the 80 MMR-SMRs included in our review, 53 (66%) provided recommendations for future MMR studies. It was more common for reviews to provide implicit recommendations (n = 31, 58%) than explicitly labeled recommendations (n = 22, 42%) to guide the use of MMR in future studies. We describe and provide examples of each theme below.
Identify MMR Design and Methodological Details
The most common recommendation made was a call for transparency and clarity regarding design decisions in empirical MMR studies. Sometimes this recommendation was framed as a suggestion to specify the MMR research design based on a particular typology. For example, a review of counseling psychology MMR studies recommended that “researchers specify the type of mixed methods research design used (e.g., sequential explanatory mixed methods design)” (Hanson et al., 2005, p. 233). This call was echoed in a review of MMR’s use in immigrant mental health studies that asserted “the chosen MMR design should be explicitly mentioned in the methods section to identify the relative emphasis of the quantitative and qualitative components” (Islam & Oremus, 2014, p. 1289).
In other instances, authors advocated for the specific design decisions to be specified, rather than depending on a preexisting design typology to capture the nuanced choices that were implemented in a study. For example, researchers were encouraged to provide additional details about “data collection, data analysis, and integration of methods, interpretation, and reporting” (DeJonckheere et al., 2018, p. 20). Providing clear details in research reports would “assist the community of interdisciplinary researchers in understanding both the research procedures (e.g., mixed methods integration) and results (e.g., theory generated from grounded theory analysis)” (Guetterman et al., 2019). Similarly, more detailed reporting of mixed methods design decisions would “allow for more comprehensive assessment of findings, which should lead to increased validity of evidence, improved research designs and ultimately, better interventions” (Brown et al., 2015, p. 831). Additionally, clearly reporting methodological decisions would allow reviewers and readers to more accurately judge the quality of the MMR study (Wisdom et al., 2012).
Provide Transparency Related to Other MMR Elements
In addition to transparency in design, authors of reviews called for other types of transparency. For example, authors encouraged use of the term “mixed methods” when publishing MMR studies in, for example, titles (Hanson et al., 2005), abstracts (Coyle et al., 2018), and method sections (Gambrel et al., 2013). Regardless of where the term “mixed methods” appears in a publication, its inclusion not only assists other researchers and methodologists in identifying MMR studies but also signals the clear intent of the authors to integrate quantitative and qualitative strands of a study. Another call for more transparency was related to clearly stating the rationale for using MMR. Across our pool of MMR-SMRs, a variety of authors recommend that empirical MMR studies should more clearly state the rationale for selecting an MMR approach (e.g., Hanson et al., 2005; McManamny et al., 2015). In particular, the selected rationale for choosing MMR “should be grounded in the respective literature” (DeJonckheere et al., 2018, p. 497). A clearly stated rationale for choosing MMR is one evaluation criterion; hence, the absence of an explicit purpose for conducting an MMR study impacts the overall assessment of the quality of the research (Gambrel et al., 2013).
Address Quality of MMR Studies
Another theme that emerged centered on the importance of ensuring the quality of MMR studies. While some reviews focus on specific elements of quality, such as citing existing MMR literature (DeJonckheere et al., 2018), others call for broad criteria for evaluating the quality of empirical MMR studies (Brown et al., 2015; Creswell et al., 2004; McManamny et al., 2015). The use of published guidelines for reporting MMR such as Good Reporting of a Mixed Methods Study (O’Cathain et al., 2008) is encouraged to increase the quality of MMR studies (Boeije et al., 2013).
Discuss Role of Paradigms within MMR Studies
The role of research paradigms and a push to address relevant philosophical perspectives is much discussed in MMR literature, including these MMR-SMRs. Several authors of the reviews in our pool add to this discussion by encouraging researchers to address paradigmatic issues by being more explicit about their paradigmatic stance (Alise & Teddlie, 2010) and attending closely to any theoretical or paradigmatic issues that arise in studies (Hanson et al., 2005). The quality of MMR can be improved when researchers “attend to the philosophical tensions inherent in mixing qualitative and quantitative methodologies and reflect on the limitations and challenges of the chosen mixed methods design” (Bishop & Holmes, 2013, p. 5).
Conduct More Research on MMR
In addition to calls for more empirical MMR studies across disciplines (e.g., Powell et al., 2008; Sahin & Naylor, 2017) and more journal space allowed for MMR studies (e.g., DeJonckheere et al., 2018; Huynh et al., 2019; Molina-Azorín, 2011), there was a clear theme recommending more MMR-SMRs. MMR scholars have an opportunity to educate their colleagues by publishing, promoting, and sharing more exemplary MMR studies (van der Roest et al., 2015). Additional MMR-SMRs, including replications of existing reviews, could help researchers develop a better understanding of mixed methods approaches across various fields of study (Creswell et al., 2004). Also, increased MMR research is needed to ascertain which forms of mixed methods research are best suited for certain fields of study (McManamny et al., 2015). Further research is needed on rigor in MMR studies and comparisons to other single-method studies (Sahin & Naylor, 2017). Meta-analyses that summarize the cumulative results of empirical MMR studies and give an overall picture of the use of MMR across disciplines are also needed (Alise & Teddlie, 2010).
Specify the Impact of MMR Studies
Reflecting on and reporting the impact and added value of MMR studies was another theme in the recommendations. One review identified “several potential avenues” for increasing the added value of MMR studies, such as the combination of different methodological approaches in both data collection and analysis as well as the use of mixed methods throughout the entire research process; yet many of these opportunities were “underutilized” (Hurmerinta-Peltomäki & Nummela, 2006, p. 454). Clearly delineated contributions of each component of an MMR study, as well as in combination, adds to the field in a more comprehensive way (Boeije et al., 2013). Additionally, impact analysis of mixed methods studies should be carried out in other fields to compare results (Molina-Azorín, 2011). Comparison of the impact of mixed method and non–mixed method studies is needed and can be accomplished in a variety of methods including interviews of current researchers in the field.
Provide Training in MMR
The final theme we identified was the need to increase training opportunities and leadership density in MMR. The evolution and growth of MMR clearly has “implications for research training and building capacity” (Cameron & Molina-Azorín, 2011, p. 268) and that increasing training opportunities is necessary to improve the implementation of MMR (Molina-Azorín, 2011). MMR-SMRs of disciplines and topics including volunteering (Paull & Girardi, 2017), immigrant mental health (Islam & Oremus, 2014), and population health (Brown et al., 2015) echoed the call for more training in MMR skills. This training may be provided through undergraduate and graduate research methods courses (Islam & Oremus, 2014), workshops sponsored by funding agencies (Plano Clark, 2010), or a mentor who could provide guidance in implementing MMR (Guetterman et al., 2019).
Limitations Discussed in MMR-SMRs
Of the 80 MMR-SMRs included in our review, slightly more than half (n = 45, 56%) of the authors described limitations of their review methods. The limitations most typically related to decisions the authors made regarding the article identification process. Issues such as databases or journals selected to search, limitations on the date range of the search, specific search terms, and other inclusion criteria were commonly noted. Authors also acknowledged that their coding protocols were limited to information that was published, therefore may not reflect the actual procedures of their reviewed studies either based on misinterpretation of available information or omission of details in the published study. Some authors also commented on the limitations of the studies included in their review; four (5%) reviews included general comments about limitations while three (4%) specifically analyzed the stated limitations.
Discussion
As the use of MMR expands into new disciplines and research topics, as well as intersects with other methodologies and approaches, the need to understand how MMR is being implemented increases. MMR-SMRs provide a vehicle to not only capture how MMR is being used in various contexts but also to inform how MMR can best be employed within and across those contexts (Guetterman & Fetters, 2018). However, little specific guidance is provided to MMR methodologists in how to conduct systematic methodological reviews. Based on our systematic methodological review of 80 MMR-SMRs, we offer the following guidance for conducting and reporting MMR-SMRs.
Recommendation 1: Label the Type of Methodological Review
As with empirical MMR studies, finding MMR-SMRs can be challenging. Reviews that we analyzed in this study described themselves as a content analysis, critical review, narrative review, scoping review, systematic review, and so on. While each of these terms has a specific meaning, they tend to be used as synonyms for studies we would label as an MMR-SMR. To facilitate identifying relevant reviews and clarifying the general intent of the review, we recommend that future methodological reviews are labeled as one of the types described in Table 1. Within these general types of reviews, authors might use an approach (e.g., general research study, MMR study) or method (e.g., content analysis, narrative review) in conducting their review, which should also be clearly noted. By explicitly and consistently labeling future reviews, methodologists and MMR practitioners will more easily be able to identify and evaluate these reviews, thereby increasing the reviews’ impact on future MMR studies.
Recommendation 2: Define Clear MMR-SMR Purpose and Research Questions
Molina-Azorín and Fetters (2016) noted the importance of defining the methodological aim of MMR-PSs, and we extend that recommendation to any type of systematic methodological review. The purpose statement provides a focus for the review and serves as a guide to help the reader understand the intent of the study. Additionally, we encourage scholars to provide research questions that align with the review’s purpose. Research questions provide an anchor for the review, shaping the methodological choices, analysis approach, and ultimately the value of the review to inform future practice.
Recommendation 3: Submit MMR-SMR to Journal that Best Aligns with Primary Audience
MMR-SMRs have the potential to appeal to a variety of audiences. Disciplinary- or topic-based methodological reviews would inherently appeal to novice MMR scholars in the respective field who are interested in learning more about MMR and how they might use it in their own work, such as mindfulness (Huynh et al., 2019) and marketing (Harrison & Reilly, 2011). These reviews tend to focus on basic MMR features, such as design typology, priority, and integration approach in order to provide an introductory overview of MMR and its use within the field. Scholars in the field of MMR would also be interested in reviews that focus on more advanced methods or in-depth methodological issues across a pool of articles such as the value of divergent findings (Creamer, 2018), MMR sampling designs (Collins et al., 2006) and identifying rigor in MMR studies (Brown et al., 2015). In order to reach the intended audience, authors should submit their review to a journal that best aligns with their primary audience. Hence, authors should consider submitting disciplinary- or topic-based reviews to a relevant disciplinary- or topic-based journal. Likewise, authors should consider submitting reviews targeted to MMR scholars to the Journal of Mixed Methods Research or the International Journal of Multiple Research Approaches. This is not to say that either approach is mutually exclusive; however, publishing systematic methodological reviews in journals mostly likely to be seen by the intended audience increases the potential of the review to impact future practice.
Recommendation 4: Justify Literature Search Procedures
We echo the call for the methods of conducting an MMR-SMR to be rigorous and transparent so that readers can evaluate the adequacy of the identification procedures, relevance of the inclusion criteria, and meaningfulness of the findings for themselves (Molina-Azorín & Fetters, 2016). In addition to detailing the procedures for identifying studies to review, authors should also describe their reasons for selecting their specific procedures. Authors should clearly specify what date range, search engines, and search terms were used and why they selected those parameters. Without clear explication of the methodological decisions and the rationale for them, the objectivity of the review can be called into question. Furthermore, the utility of reviews lacking in methodological detail is limited in terms of modeling for other researchers how they might conduct such a review.
Databases
It is not our intention to proclaim the best strategy for identifying studies for MMR-SMRs; on the contrary, search strategies need to fit the overall purpose of the study. However, our review identified trends that authors of future methodological reviews should consider when deciding the parameters of their review. In general, MMR-SMRs should use a variety of search engines to identify potential studies for review. Each search engine will return slightly different results, so authors are encouraged to be knowledgeable about those differences and use search engines that most closely align with their overall purpose and research questions.
Search Terms
Although Creswell and Plano Clark (2018) have consistently recommended using variations of “mixed methods” and combinations of quantitative and qualitative terms to identify MMR studies, authors of MMR-SMRs in our pool most commonly limited their search terms to “mixed methods.” As noted by Molina-Azorín and Fetters (2016), authors who conduct systematic methodological reviews should use caution when limiting their search terms to only “mixed methods,” especially if their date range predates the generally accepted use of that term. However, there are certainly cases where authors provide justification for limiting their search terms, such as identifying trends across studies explicitly conceptualized as MMR (e.g., Plano Clark et al., 2015). Therefore, using combinations of quantitative and qualitative terms would have been inconsistent with that study’s purpose. We encourage authors of systematic methodological reviews to use a variety of search terms when their purpose is to explore or understand the use of MMR within a field or methodological issue.
Time Span
Given that the mean number of years included in MMR-SMRs was 11 years for reviews with a restricted date range as well as those with an unrestricted date range, it would be reasonable for future reviews to consider including studies at least across a 10-year span. As with other methodological decisions, this should not be viewed as a hard and fast rule. For example, if an author were examining the impact of a previous methodological review, choosing dates from the end of the previous review to the present (e.g., Coyle et al., 2018; Harrison & Reilly, 2011) would be an appropriate exception to the 10-year recommended time span.
Prevalence Rates
As noted previously, MMR-PSs are a subset of MMR-SMRs. For MMR-PSs, justifying the journal(s) selected, date range, and definition of MMR are key details that are necessary to understanding the context of the findings (Molina-Azorín & Fetters, 2016).
Recommendation 5: Specify Strategies Used to Finalize the Pool of Studies Analyzed
In addition to describing the search procedures, authors should also clearly specify any procedures used to refine the initial pool of articles. When studies are determined to be outside the scope of the review, the reason for their removal should be noted. We encourage authors to include a PRISMA diagram (Moher et al., 2009) or other visual depiction to represent the stages of their identification process and clearly account for why and how many articles were removed.
Our analysis of the 80 reviews in our pool did not yield a clear recommendation for how many empirical studies are adequate to conduct a methodological review. The number of empirical MMR studies analyzed within an individual review ranged widely from five to 332. Hence, determining the adequacy of the number of empirical studies to include in an MMR-SMR is not that more is better and fewer is weaker; the context and content of the scope of the review will necessarily guide that determination.
Recommendation 6: Clearly Describe Analysis Process
As with many of the other features in MMR-SMRs, how the empirical studies are analyzed will vary based on the stated purpose and research questions of the review. Nevertheless, we recommend that authors of MMR-SMRs clearly describe their analysis process. This includes defining not only what methodological elements are being analyzed (e.g., MMR design, rationale, sampling) but also the process used to conduct that analysis. Reviews with more than one author should clearly state how their empirical studies were analyzed across the research team and their process for addressing inter-rater agreement. The precise approach to addressing interrater agreement should be driven by the review’s purpose and research questions and clearly described in the review.
Among the MMR-SMRs in our pool, there were a variety of MMR issues analyzed across the empirical articles. Reviews typically addressed MMR designs, MMR definitions and MMR rationales. While we agree that these are important elements to report in an MMR-SMR, we encourage authors of future MMR-SMRs to also include more advanced or conceptual features in their analyses. For example, authors of disciplinary- or topic-based systematic methodological reviews in our pool evaluated the quality of MMR empirical studies (e.g., Bishop & Holmes, 2013; O’Cathain et al., 2008; Wisdom et al., 2012); discussed paradigms (e.g., McManamny et al., 2015; van der Roest et al., 2015); and identified the value-added contributions of inferences generated as a result of using an MMR approach (e.g., Hurmerinta-Peltomäki & Nummela, 2006; Sahin & Naylor, 2017). Conducting more complex analyses of methodological features within MMR-SMRs increases the value of their contributions to our understanding of how the use of MMR is evolving.
We also encourage the expansion of MMR-SMRs that focus on methodological features to frame their review. Reviews in our pool that focused on methodological intersections such as the use of longitudinal designs (Plano Clark et al., 2015), case studies (Guetterman & Fetters, 2018), grounded theory (Guetterman et al., 2019), community-based participatory research (DeJonckheere et al., 2018), and transformative approaches (Sweetman et al., 2010) have made valuable contributions to the field of MMR and our understanding of the nuanced contributions of these approaches.
Recommendation 7: Provide Complete Details of Analysis Results (i.e., Coding Table)
We further recommend that authors of MMR-SMRs provide complete details of their analysis. Most reviews in our pool provided a table summarizing the findings of their coding, noting what percentage of the articles included those features. However, only a handful of reviews provided a complete coding table that shows how each individual study reviewed was coded for the methodological elements being reported, so that these details could be readily studied by readers. We argue that reviews published without a complete coding table in the publication (e.g., DeJonckheere et al., 2018), as an online supplementary material (e.g., Guetterman et al., 2019), or on request (e.g., Molina-Azorin, 2012), minimize the potential impact on their field, as well as the field of MMR, because it is difficult to evaluate the validity of their findings when the coding details are not available. Furthermore, the lack of a complete coding table limits the ability of readers to explore specific MMR elements within the scope of the review when there is no way to link individual empirical studies to those elements. We further encourage authors of MMR-PSs to fully report their findings so that it is clear whether the percentage of MMR studies is based on the total number of empirical studies only or all articles (including editorial, methodological pieces, book reviews, etc.) and to provide sufficient information (e.g., a table) in order to compare prevalence rates across disciplines.
Recommendation 8: Illustrate MMR Concepts With Details From Reviewed Studies
Most of the reviews in our pool provided descriptive statistics to summarize the presence or absence of MMR features and design decisions. While this type of summary captures the broad trends across a pool of studies, it fails to bring the studies to life for the reader, thus minimizing the review’s ability to inspire future MMR studies in the relevant discipline or topic. To be effective, examples should provide not only enough detail to clearly illustrate the methodological concept but also provide enough details regarding the content of the study so that scholars in that field or discipline have a concrete example for how they might use MMR in their own work. For example, in describing the use of MMR research questions in a pool of MMR studies in mindfulness research, Huynh et al. (2019) quoted the sole MMR research question from their pool of empirical studies and then described how that research question foreshadowed the supporting role of the qualitative strand and a merging approach to integration.
Recommendation 9: Describe Limitations of Review Methods
We recommend that authors of MMR-MSRs identify possible limitations in their review process, as they can affect the results of a review. For example, a limitation of our study, as with any systematic review, was our search procedures. Using additional databases and different search terms may have identified additional examples that might yield different results. In particular, we did not use any databases geared toward the health sciences, focusing instead on education and social sciences. However, by using a combination of systematic and nonsystematic search approaches, we were able to identify 80 examples reviews which provided ample data for our research purpose, a full third of which related to health domains. Our review was also limited by the information published in the reviews. Authors of MMR-SMRs may have conducted additional procedures that were not reported in their publications, which would not be reflected in our study.
Recommendation 10: Make Explicit Recommendations to the Field of MMR
As noted by Molina-Azorín and Fetters (2016), the value of MMR-MSRs is the authors’ interpretation of how their study can inform the field of MMR. We recommend that authors of MMR-MSRs make explicit recommendations for how MMR should and/or could be used in their discipline or field. Explicit recommendations provide clear guidance for future MMR studies, thus increasing the value of contribution of the review. Examples of recommendations are presented in the results section above; however, recommendations should not be limited to the types of recommendations proposed in other MMR-SMRs. Authors of MMR-SMRs should also strive to specify recommendations that are unique or novel to the context of their review. In particular, we encourage authors to propose recommendations that highlight the methodological contribution of their review.
Checklist for Publishing a Mixed Methods Research Systematic Methodological Reviews
Based on our recommendations for conducting MMR-SMRs, we offer this checklist (Table 7) not only to guide the development of future MMR-SMRs but also to assist peer reviewers in evaluating MMR-SMRs for publication.
Checklist for Publishing an MMR-SMR.
Note. MMR = mixed methods research; SMR = systematic methodological review; PRISMA = Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
Conclusion
Mixed methods research systematic methodological reviews serve not only to describe how MMR is used within a discipline or research topic but also to elevate our understanding of the nuanced ways in which MMR is evolving across those applications. Our review highlights the variety of approaches to conducting MMR-SMRs across diverse fields of study and methodological foci. Based on these findings, we propose terminology to clarify the types of MMR-SMRs as well as recommendations that serve as a framework to guide the development and review of future MMR-SMRs. We believe these contributions will serve to elevate the accessibility and quality of MMR-SMRs and thus will increase their value in guiding the continued development of MMR research.
We would like to invite other MMR scholars to contribute to this emerging line of research. For example, our study could be replicated using different search engines and different search terms to see what new MMR-SMRs would be identified and how they might be different from those already in our pool. We are also interested in analyzing how mixed methods research is defined by authors of MMR-SMRs and the extent to which certain types of studies are being included or excluded from MMR-SMRs. As suggested by one of the reviewers of this study, we are interested in conducting a qualitative study of authors who conduct MMR-SMRs to learn more about the context of their work and how they view their contributions to the field of MMR, similar to the author interviews conducted by Molina-Azorín and Font (2016), O’Cathain et al. (2007), and Zhang (2014). We encourage the emerging line of MMR-SMRs based on nonpublicly available reviewer comments (e.g., grant proposals, Guetterman et al., 2019; submitted manuscripts, Wilkinson & Staley, 2019), as they provide important feedback to researchers seeking to fund or publish their work. We also believe a study of MMR educators and their use of MMR-SMRs as pedagogical tools would make an important contribution to the MMR field.
Supplemental Material
Conducting_MMR-SMR_online_supplement – Supplemental material for Conducting Mixed Methods Research Systematic Methodological Reviews: A Review of Practice and Recommendations
Supplemental material, Conducting_MMR-SMR_online_supplement for Conducting Mixed Methods Research Systematic Methodological Reviews: A Review of Practice and Recommendations by Michelle C. Howell Smith and Pamela Shanahan Bazis in Journal of Mixed Methods Research
Footnotes
Acknowledgements
Special thank you to Mykesha Jackson, Department of Child, Youth and Family Studies, University of Nebraska–Lincoln, for her assistance in developing the initial codebook for this study.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
