Abstract
As mixed methods continues to grow as a discipline, work to define what constitutes quality mixed methods research has become an emergent conversation. While progress has been made in this area, there has been some debate as to what quality entails and how to achieve it. This article contributes to mixed methods by highlighting the importance of rigor as an interdisciplinary baseline for quality evaluation and proposes a rigorous mixed methods framework. This framework is then applied to the management studies literature to give insight into a literature base where mixed methods research is still relatively nascent. Findings give examples of current practices in management studies as well as an example of how the rigorous mixed methods framework can be operationalized.
While combining qualitative and quantitative methods constitutes a long-standing research practice (Greene, 2006; Pluye et al., 2009), the conceptualization of such practices as mixed methods research is a relatively recent occurrence (Tashakkori & Teddlie, 2003). However, as mixed methods research continues to grow as a discipline, including more than 31 books devoted primarily to mixed methods research (Onwuegbuzie, 2012), it is important to review usage patterns to advance an understanding of how scholars are utilizing the methodological approach. The methodological aim of this article is to identify a framework for rigorous mixed methods (Rigorous Mixed Methods) research and then apply that framework to the area of management studies to illustrate how mixed methods are being utilized in a discipline where the formal adoption of mixed methods is relatively recent. The first phase of this study focuses on the conceptualization, operationalization, and reporting of rigor rather than a discussion of quality. While rigor and quality are often correlated in academia, there are idiosyncrasies across various disciplines and regions as to what constitutes high-quality and low-quality work. For example, researchers in Europe might see qualitative research as more valuable than researchers in the United States. In the publishing process, it is often possible for an imperfect article with a significant contribution to be seen as a high-quality article, while highly rigorous work with a relatively small contribution is seen as holding lesser value. This reflects concern about quality where it is subjective to individuals and disciplines, and discussions of quality often become a compensatory process where shortcomings in one area can be overcome by excellence in another (and vice versa). By focusing on the rigor in the mixed methods research process and the rigorous reporting of said research, this article advance the discipline of mixed methods research in an objective and clear manner. To demonstrate the Rigorous Mixed Methods framework, we review and evaluate the rigor of recently published research in management science and offer descriptions of best practices related to mixed methods rigor. Furthermore, a content analysis of a 6-year sample of published management science articles is presented to identify a representative sample of articles that employ a mixed methods approach. Those articles are then analyzed using the Rigorous Mixed Methods framework to highlight the characteristics of rigor employed by management scholars in conducting mixed methods research. In doing so, we address issues related to quality and rigor assessments in mixed methods management research practices, while discussing how mixed methods is currently being used by management scholars as well as investigating current trends and best practices.
We begin this article by discussing definitions of mixed methods research and introducing our framework for Rigorous Mixed Methods. We then present a content analysis of mixed methods articles across four highly ranked management studies journals over a 6-year span between 2009 and 2014. To illustrate further the application of rigor in management science, we then provide two exemplars of mixed methods research in management science. Our final discussion highlights the elements of rigor seen most commonly in our subset of articles and suggests recommendations for researchers/trainees conducting and reporting mixed methods in management studies, as well as for reviewers/editors who are assessing mixed methods manuscripts, papers, and grant applications.
Definitions of Mixed Methods and Mixed Methods in Management
To begin, mixed methods research should not be confused with “mixed model” research, wherein statistical analysis is conducted of fixed and random effects in a database. Mixed methods research should also not be confused with “multiple methods” research in which multiple forms of qualitative data (e.g., interview and observations) or multiple forms of quantitative data (e.g., surveys and experiments) are collected and analyzed. Several definitions exist as to what constitutes “mixed methods.” In this article, we utilize the following definition based on an analysis of definitions used by leaders in the field of mixed methods research: Mixed methods research is the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purpose of breadth and depth of understanding and corroboration. (R. B. Johnson et al., 2007, p. 123)
The core assumption of this approach is that when a researcher, or team of researchers, combines statistical trends (quantitative data) with lived experiences (qualitative data), the collective strength of both data types provides a better understanding than one data type alone (Bryman, 2006; Greene et al., 1989). Although both data types are coalesced into one study, it does not mean that the scope of each type is diminished in any way. That is, the rigorousness of each monomethodological approach should be maintained as part of the mixed methods approach. As there are guidelines for assessing the rigor in any monomethod design, we now introduce guidelines with which to evaluate the rigor of mixed methods research. This discussion allows for both mixed methods researchers and reviewers to understand and assess methodological rigor in articles that employ a mixed methods approach.
A history of the use of mixed methods in management studies can be traced back to the Hawthorne studies conducted in the early 20th century. The Hawthorne studies investigated principles of employee behavior and emphasized experiment and extensive interviewing and observational data. These studies of social relations among employees highlighted the need for unobtrusive methods of inquiry in management research (Tashakkori & Teddlie, 2003). A more detailed account of the history and pioneers of mixed methods research in business are offered by Molina-Azorin and Cameron (2015).
The complex nature of human behaviors often requires management scholars to understand behaviors in both breadth and depth, a task for which mixed methods research is especially well suited. In addition, management studies often include varying contexts with complex open systems and the management of people at different levels within the organization. Molina-Azorin and Cameron (2015) suggest that mixed methods research can add value in management research because it offers the opportunity to yield insight regarding both process (qualitative) and outcomes (quantitative)—or individual-level (qualitative) and firm-level (quantitative) domains. Furthermore, Hurmerinta-Peltomaki and Nummela (2006) found that in the field of business, mixed methods add value by increasing validity in the findings, informing the collection of the second data source, and assisting with knowledge creation. Gibson (2017) suggested the concept of “methodological fit” (p. 6) between method and research question as being a potential driver of the increase in impact and rigor of a mixed methods study, suggesting that authors should analyze their research question, the current state of the literature, as well as their intended theoretical contributions when deciding to do mixed methods work. Gibson goes on to propose that mixed methods research tend to offer four main values to researchers: elaboration, generalization, data integration, and interpretations. She proposes that if these values are pursued by researchers, they give an opportunity to develop further insight into an area than a single monomethod presentation. This perspective is supported by Molina-Azorin (2011) who found that mixed method articles had a higher impact on the management discipline. While the value of mixed methods research is generally reaching consensus, the management discipline has the opportunity to formalize its approach to mixed methods. Cameron, Sankaran, and Scales (2015), in a review of the product management literature, found that while the prevalence of mixed methods articles had increased, the reporting of the methods had not kept pace with other disciplines. In concert with advancing the Rigorous Mixed Methods framework, this article seeks to close that gap.
Defining Rigorous Mixed Methods
Understanding of what constitutes methodological rigor is a key step in developing the Rigorous Mixed Methods framework, and a brief discussion of quality in mixed methods is included here to facilitate this definition. In their recent article, Hong and Pluye (2018) define methodological quality as being “concerned with how a study is conducted. It is usually related to the construct of trustworthiness: Is a study good enough for the results to be trustworthy?” (p. 5). They go on to further state that this “trustworthiness” is generally assessed based on the methodologies used and the potential biases that might exist in a given study. Similarly, Leech et al. (2010) proposed a validation framework that included a combination of both qualitative and quantitative validity to assess mixed methods validity, which they posit will lead to higher quality mixed methods articles. Central to the validation framework is the concept of legitimation, drawn from Onwuegbuzie and Johnson (2006) and discussed using words such as “trustworthy,” credible,” and “dependable.” Another recent study included interviews with researchers across several disciplines who had written methodological pieces on mixed methods research (Fàbregues et al., 2018). While these authors identified several criteria that appeared to hold across academic disciplines, they note that any discussion of quality “must include the views of researchers regarding the very nature of criteria generation, including the approach to appraising quality or even whether a consensus on criteria is appropriate to the field of MMR” (Fàbregues et al., 2018, p. 425). This presents a key problem for researchers, editors, and reviewers. If all quality must be viewed through the lens of the discipline in which the study is conducted, and if key aspects of quality are related to a subjective concept like “trustworthiness,” how can an integrated definition of quality be created that will hold across disciplines? Fàbregues et al. (2018) make headway on this question, as their results show that there are at least two perspectives currently held toward quality, and that these perspectives are related to the discipline of the researcher interviewed. Rather than expand this discussion of quality, we narrow our focus to solely that of rigor in an effort to provide a foundation for further quality discussions.
Current best practices for reporting rigor (although often used synonymously with quality) are perhaps best exemplified by O’Cathain et al. (2008), who proposed a reporting paradigm called the good reporting of a mixed methods study (GRAMMS). The GRAMMS framework asks authors to report a justification, the design type, the components of each mono method, the data integration of the study, limitations of the study, and insights gained from the mixing of data. O’Cathain (2010) has also written about developing a framework for quality in mixed methods research, which included a discussion of rigor, but focused primarily on judging the overall quality of a research piece. O’Cathain discusses rigor as a subcomponent of quality, and rigor is mentioned in several of her quality domains, yet a formal set of rigorous steps is not presented for authors. Further discussion of the concept of rigor in mixed methods can be found in the area of health services (Wisdom et al., 2012), where authors propose a key point in the rigor debate—that quality and rigor are separate—and further suggest that journal editors consider publishing guidelines for rigor in mixed methods research. The following Rigorous Mixed Methods framework answers that call. As suggested by Wisdom et al. (2012), rigor is primarily concerned with both the actions of the researcher (i.e., the steps taken in the scientific process) and the reporting mechanisms used to describe those steps to the reader, providing a better understanding of exactly what the researchers did during their study. Rigor in this case is not designed to replace quality but, rather, to facilitate the review of quality in a manuscript. By definition, rigor does not necessarily make an article high quality, but properly utilized rigor does allow for a reader, editor, or reviewer to judge the quality of the article. The first major contribution of this work is to clearly delineate between quality and rigor and provide guidelines for conducting rigorous research. Again, as noted previously, the Rigorous Mixed Methods framework is not intended as a substitute for quality but, rather, a framework that allows quality to be assessed. The second major component of the Rigorous Mixed Methods framework is that it conceptualizes rigor as a continuum in mixed methods. This continuum, currently operationalized into categories of high, medium, and low for most criteria, adds a more nuanced evaluation of a project to existing reporting checklists. The next section of the article describes the development and components of the Rigorous Mixed Methods framework.
Our Rigorous Mixed Methods framework extends prior work on mixed methods reporting by highlighting a process for conducting and reporting research according to primary and advanced criteria for rigor. Rigorous Mixed Methods incorporates aspects of prior mixed methods reporting guidelines, but it goes further by delineating author tasks when conducting mixed methods research as well. In addition, the Rigorous Mixed Methods framework presented here provides prescriptive advice on how to conduct mixed methods research and includes a conceptualization of what constitutes high, medium, and low rigor for each of the Rigorous Mixed Methods categories. Our framework is informed by criteria utilized by one of the authors as evaluative criteria for articles submitted to the Journal of Mixed Methods Research during his tenure as editor of the journal.
Our discussion of Rigorous Mixed Methods is organized into two categories—primary and advanced elements. The four primary elements represent what we describe as the core characteristics of mixed methods research. The first element describes rigorous data collection of each data strand (i.e., qualitative and quantitative). The second element describes rigorous data analysis of each data strand (i.e., qualitative and quantitative). The third element describes the “integration” or mixing of both data strands. This mixing is the cornerstone of mixed methods research, as the premise of mixed methods is that the integration of data leads to more than the sum of its parts, and potential methods of data integration will be discussed later on in this article. The fourth primary element describes the use of a specific mixed methods design type.
While all primary elements of a mixed methods research design are important, a brief discussion of design types is useful here to set the stage for our content analysis. Mixed methods design types are as central to conducting mixed methods projects as they are in projects labeled as “experiments,”“surveys,” or “ethnographies,” because they help scholars understand and convey how the studies are presented. The three basic mixed methods design types highlighted in this study are exploratory sequential designs, explanatory sequential designs, and convergent designs (Creswell & Clark, 2017). In exploratory sequential designs, researchers first collect qualitative data, analyze the qualitative data, and then build on the qualitative data for the quantitative follow-up. In explanatory sequential designs, researchers first collect and analyze quantitative data, then build on those findings in a qualitative follow-up, which seeks to provide a better understanding of the quantitative results. In convergent designs, researchers collect and analyze quantitative and qualitative data with the intent to merge the results of the quantitative and qualitative data analysis.
Beyond these four primary elements are advanced procedures or techniques that have developed in mixed methods research (Creswell & Clark, 2017). These procedures add to overall mixed methodological rigor and include (1) providing a discussion of the aims and purpose for the use of mixed methods and (2) including elements of the writing that aim to promote the usage of mixed methods research through language and terminology. Presenting the aims and purposes involves providing a clear rationale for conducting a mixed methods study, including a mixed methods research question, and discussing the value of mixed methods research. Presenting the elements of writing involves referencing mixed methods literature, using of joint displays to show integration, and including mixed methods in the title. Examples of the varying rankings of each element are outlined in Table 1.
Rigorous Mixed Methods (Rigorous Mixed Methods) Coding Scheme.
In sum, these elements highlight the methodological rigor associated with various sections within a manuscript. Next, we investigate how mixed methods rigor in management studies is reported. Specifically, we conduct a content analysis of the management studies literature to explore the ways in which scholars utilize the methodology.
Methodology
In this article, mixed methods articles were reviewed across four highly ranked management studies journals over a 6-year span: the Academy of Management Journal, Administrative Science Quarterly, Strategic Management Journal, and the Journal of Management. These journals were chosen to provide data from eminent journals, and it is assumed that they represent trends in best practice research. We feel that 6 years represent a sufficient period of time to assess trends of recently published research. Next, we conduct a content analysis investigating the current state of mixed methods rigor in management studies. This analysis is similar to a mixed studies review as defined by Pluye et al. (2009)and Hong and Pluye (2018) 1 ; however, rather than focusing on a topic and presenting both quantitative and quantitative studies, we focus in on the methodological makeup of each article in the data set. Content analysis is an observational technique that allows for a systematic evaluation of recorded communications (Kolbe & Burnett, 1991). In the current investigation, the recorded communication of interest are the four selected journal publications. For the purpose of coding, articles were required to contain qualitative research centered on primary data collection in nonnumerical form (words, images, symbols, etc.) and quantitative research centered on data collection in numerical form. Also, as part of our exclusion criteria, articles that quantified qualitative data (also called “quantitizing”), or transformed quantitative data in qualitative data (also called “qualitizing”), are the topic of debate among mixed methods scholars in terms of mixed method distinction; to prevent obscuring the central focus of this article, these articles were omitted.
Our coding procedures followed a two-part process. First, articles were identified as including the collection and reporting of both qualitative and quantitative data in some form. We were incredibly inclusive in this portion of the data collection in an effort to identify as many articles as possible. In Phase 2, articles were coded according to the criteria outlined in Table 1 and identified as having an overall level of rigor. Most of the articles in the data set fell into the low level of rigor column entirely. Technically, many of these low-level studies would not be considered true “mixed methods studies” due to their lack of data integration. However, they were included as a way to highlight the varying ways in which scholars are collecting and analyzing both qualitative and quantitative data in management studies. To identify an article’s overall level of rigor, we identified each article’s level of rigor across the Rigorous Mixed Methods elements. Most articles fell nearly entirely within one level, but several articles were cross-classified with at least one Rigorous Mixed Methods element in the nonmajority (i.e., low, medium, or high category). For these articles, we created a cross-level classification scheme (e.g., low–medium and medium–high). Most articles tended to have a certain level of rigor, and no article reached across all three levels of rigor. This is likely due to the requirements for an article to be classified as having a high level of rigor. All disagreements in coding were discussed and resolved by the authors.
Findings
There were 195 articles, out of 1,164 empirical articles, that incorporated the inclusionary element of having collected both qualitative and quantitative data. Each article was analyzed using the coding scheme outlined in Table 1, resulting in the data set of articles as shown in Table 2.
Rigorous Mixed Methods Inclusionary Rankings.
The breakdown across journals showed 72 articles in the Academy of Management Journal, 15 in Administrative Science Quarterly, 73 in the Strategic Management Journal, and 35 in the Journal of Management. Each article was classified as having a low level of Rigorous Mixed Methods to a high level of Rigorous Mixed Methods using the coding criteria presented in Table 1. Of the 195 articles coded combining both qualitative and quantitative data, 128 were classified as having low rigor. The most common reason for having a low level of rigor was a complete lack of emphasis on the mixed portion of the methodology, despite reporting some component of both qualitative and quantitative data collection as part of the research study. These articles were often quantitative articles that referenced depth interviews as part of scale development or pretesting but did not report any aspect of the qualitative work other than to potentially reference it when discussing how their survey items were created. This type of study nominally fits the “low” level for each criteria in Table 1. By R. B. Johnson et al.’s (2007) definition, this low level of rigor would not meet the standards for being classified as a mixed methods study. However, we included these low-level articles to highlight both the frequency of articles that combine both data types and the opportunity to enhance methodological rigor. One step up from the low rigor articles were the low–medium rigor articles. Of the 195 mixed methods articles classified, 20 were qualified as having a low–medium rigor level. To be classified as a low–medium article, an article had to perform at a medium level for at least one of the criteria in Table 1. For example, a scale development article might report having conducted depth interviews then nominally report and analyze the qualitative data collected. These types of articles were relatively rare; however, this may be due to researchers making choices about what to include and exclude in a study when subject to length requirements. It is important to note that these articles with lower levels of rigor should not be viewed as poor scholarship, as they present a methodology that answers specific research intentions. They merely are not viewed as rigorous in mixed methods terms—something that was most likely outside of the goals of the authors. By nature, mixed methods articles have more involved methods and data reporting requirements than a monomethod study, and this may have resulted in one type of data being removed or de-emphasized during the review or editing process. The next tier of mixed methods rigor involved medium rigor articles, of which 28 were found out of 195 articles. For an article to be classified as medium, it had to meet the majority of the criteria in Table 1 at the medium level.
Of the articles remaining, 19 were classified as either medium–high or high mixed methods rigor. For an article to be classified as medium–high, it had to meet all criteria in Table 1 at the medium level and have at least one criteria at the high level. Only 13 articles were classified as medium–high. To achieve a high level of rigor, an article had to meet most of the criteria in Table 1 at the high level. Of the 1,164 articles reviewed, only six were labeled at this level of rigor. These 19 medium–high and high-level articles have been listed in Table 3 for further review. Table 3 highlights how mixed methods is currently being conducted in management studies by listing the research topic area, journal of publication, mixed methods research components, and qualitative and quantitative components. This table is designed to provide insight into a developing mixed methods research discipline as well as to provide insight into management scholars seeking to conduct mixed method studies. From a purely mixed methods standpoint, it can be valuable to assess how other disciplines adopt and utilize mixed methods protocols. To facilitate this, we conduct an in-depth discussion of the trends within the management discipline. Researchers may also use this table as a starting point to see what types of work have been done in their areas of study. Last, Table 3 gives further insight into the state of mixed methods research in management. This presentation is similar to that of J. S. Johnson (2015) and allows for scholars to easily compare existing articles with the criteria of Rigorous Mixed Methods suggested in this article.
Rigorous Mixed Methods in Management Journals.
Note. SMJ = Strategic Management Journal; ASQ = Administrative Science Quarterly; AMJ = Academy of Management Journal; MM = mixed methods; Y = yes; N = no; Quan = quantitative; Qual = qualitative; CEO =chief executive officer; R&D = research and development; CFA = confirmatory factor analysis; GLS = generalized least square; ANOVA = analysis of variance; ANCOVA = analysis of covariance.
The articles in Table 3 can first be delineated by research area and publication outlet. While all articles are in management journals, two articles focus on learning within the organization, while two others focus on innovations. The articles labeled as high or medium–high are primarily drawn from Academy of Management Journal (12) and Strategic Management Journal (five), although Administrative Science Quarterly had two articles in the data set.
The most common mixed methods design type was the Exploratory Sequential design, wherein a qualitative data collection was followed with a quantitative data collection (13/19). Other design types included Convergent designs, where both qualitative and quantitative data were collected simultaneously (5/19), and one multiphase design, which included an initial qualitative data collection, a subsequent quantitative data collection, and a final qualitative follow-up.
The qualitative components of these mixed methods articles were primarily composed of depth interviews with research subjects; however, two articles used at least some form of observational data collection and analysis. Quantitative analyses were generally well reported and covered a wide range of techniques; most common were descriptive analyses with some form of correlation or regression done to investigate relationships within the data.
To further the investigation of Rigorous Mixed Methods in management research, the following section identifies and discusses examples of the mixed methods research elements that have been drawn from our data set. This examination of individual elements is then followed by a more detailed discussion of two articles that serve as best practice exemplars of Rigorous Mixed Methods management articles.
Illustrating Rigorous Mixed Methods
Several articles were identified as containing multiple elements of Rigorous Mixed Methods. Although Table 1 offers a cursory description of Rigorous Mixed Methods elements, this section offers a more detailed discussion of how the various elements of mixed methods rigor are methodologically reported in the management literature. The reporting below also includes brief discussions of mixed methods techniques, providing management researchers with a clear understanding of the discussion. While both quantitative and qualitative elements of mixed methods research are important, an in-depth discussion of them individually is beyond the scope of this article. We begin by first discussing the element of mixed methods research designs, followed by a brief discussion of data integration techniques.
Though none of the articles specifically identified a mixed methods research design, the exploratory sequential approach was the dominant design type employed by organization scholars. This is not surprising, as others have found this to be the dominant design type in business research (Harrison, 2013; Harrison & Reilly, 2011).
Figure 1 shows the three basic mixed methods design strategies and the rationale an author might use for implementing one of these design choices. For Convergent designs, a mixed methods research question might be, “To what extent do the quantitative and qualitative results agree?” For exploratory sequential designs, authors might ask, “To what extent can the qualitative findings be generalized in a quantitative setting?” and for the explanatory sequential design, the authors might ask, “To what extent can the qualitative findings help explain the quantitative results?” These are quite simplistic as they are absent in any research context but can easily be put into use. For example, if a research question requires the development of a new survey instrument, a researcher can incorporate an exploratory sequential design (Creswell and Clark, 2017). The exploratory sequential design is especially suited to scale development as it allows authors to develop their items from the qualitative data collection phase. These items can then be evaluated quantitatively. Each design type should be chosen with regard to the research question of interest, as each rationale suggests formal research questions that may be examined. Convergent designs, for example, work quite well for comparison research questions where the findings from the different research types can be investigated for similarities and discrepancies. This list of research questions is not intended as an exhaustive list at this time but merely as a guide for authors to understand how these designs may be useful when crafting mixed methods research questions (Creswell & Clark, 2017).
Rigorous Mixed Methods Elements by Design Type.
While having a mixed methods research design can help identify and guide a study, the way the data are mixed is also important. Data integration refers to how the qualitative and quantitative results are brought together in a mixed methods study (Creswell & Clark, 2017). It is the point of interface between the two data strands (Morse & Niehaus, 2009). This point of interface can occur in the planning stages of research, during analysis, or at the conclusion of research for interpretation. Strategies for integrating the data are connected to the three basic designs and include (1) merging (convergent design), (2) explaining (explanatory sequential design), or (3) building (exploratory sequential design). Merging strategies in convergent designs have also been described as varying forms of data triangulation (see Turner et al., 2017, for a more detailed discussion). The latter two strategies are sometimes collectively considered “connecting” strategies for integration. In our data set, data integration most often followed standards associated with the design type utilized. That is, they were sequentially linked when discussing how the qualitative data were used to develop quantitatively testable theories or how the qualitative data helped explain aspects of the quantitative results. However, data integration can be achieved in various ways. For example, Almandoz (2012) discussed how mixing occurred at multiple stages of the research process: This mixed methods design gains from the strengths of its component parts. Archival evidence is unobtrusive and objective, while interviews add greater depth of interpretation that infuse the previous analysis with rich context. Qualitative support was critical in guiding the hypothesis generated, the controls used, and the interpretation of the [quantitative] results. (p. 1390)
This mixing of data, especially in the data analysis section of the article, allows for stronger overall analysis and interpretation by the researcher. A qualitative study may do a strong job of describing an individual’s lived experience, while having poor generalizability. A quantitative study may do a good job of describing an “average” person while describing no individual specifically. The mixed methods paradigm allows for researchers to accomplish both of these tasks. This mixing of research perspectives allows for the best of both monomethod perspectives to be included in a single study.
Having discussed the primary elements of Rigorous Mixed Methods, we will now discuss the advanced elements that may contribute to an article’s mixed methods rigor. The first advanced element involves the discussion of the aims and purpose for conducting mixed methods research. This is important because it offers evidence for the appropriateness of utilizing a mixed methods approach and includes discussing the rationale and value of mixed methods research, and the inclusion of a mixed methods research question. Several articles offer clear rationales and discussions of value for conducting mixed methods research. For example, the purpose statement created by Aime et al. (2014) states, “By combining a qualitative interview-based study and a quantitative, laboratory-based study, we were able to improve the validity of our study by countering the limitations and trade-offs inherent in each method” (p. 346). It is recommended that authors identify both a general rationale for using mixed methods and a more specific rationale related to the different mixed methods design types (Creswell & Clark, 2017). This will often reflect a need for approaching a research problem from multiple perspectives. Two studies in our data set included mixed methods research questions that address the integration, calling attention to both data strands (see Plano Clark & Badiee, 2010, for a more detailed discussion of mixed methods research questions). For example, Detert and Edmondson (2011) discussed goals in connecting their two studies.
In Study 2, we examine the generalizability of the implicit theories identified in Study 1. Our goal is this study was not to develop an exhaustive taxonomy of all self-protective implicit voice theories (hereafter, simply “implicit voice theories”), but rather to confirm that those identified in Study 1 are not idiosyncratic to a single organization and are common enough to merit subsequent investigation. (p. 462)
An additional aspect of the elements of writing includes the discussion of the value of mixed methods. Mixed methods scholars are not often explicit about stating the value of the mixed methods approach (Creswell & Clark, 2017). However, in our data set, McDermott et al. (2009) specifically discussed the value of the mixed methods approach in their study of a firm’s ability to upgrade their products. The authors state, Although we do not claim to present definitive, linear causality, our research design combines the strengths of comparative qualitative and statistical analyses to capture configurative causation—the plausibility of certain policies’ reshaping the organizational and institutional factors that significantly impact firm-level product upgrading. (p. 1271).
This presentation of value represents an understanding of mixed methods and adds rigor as a discussion of the aims and purpose for utilizing the methodology.
The writing elements involve the inclusion of a mixed methods title, including joint displays and citing mixed methods research. The title provides the focus for the entire project, and it is recommended that mixed methods title include the words “mixed methods” to denote the methodology being used (Creswell & Clark, 2017). Some may debate the importance of including a mixed methods title; however, we feel it is important as a signaling device for reviewers, readers, and editors of the type of research to follow, especially in disciplines where the technique is not well-known. In addition, the title should use neutral language, thus words that convey a qualitative leaning should be avoided (e.g., explore, meaning, or discovery) as should words that convey a quantitative leaning (e.g., relationship, correlation, or explanation). In our data set, it was common for authors to cite methods literature related to qualitative or quantitative methods. However, it was rare for them to cite mixed methods literature.
Another writing element is the inclusion of a joint display. A mixed methods joint display arrays the quantitative and the qualitative results together for a comparison in a table or a graph (Creswell & Clark, 2017). Joint displays are valuable in mixed methods research because they can assist both the researcher and the reader in thinking about the integration of qualitative and quantitative data. Joint displays may facilitate the cognitive process involved with mixing and adding perspectives. For sequential designs, joint displays link qualitative and quantitative findings by showing how the results from one phase proceeded to data collection in the subsequent phase. In an explanatory design, the researcher determines what quantitative results need further explanation and then conducts a qualitative follow-up phase (Creswell & Clark, 2017). Similarly, in an exploratory sequential design, themes, codes, and quotes may be useful to design items, variables, and scales (Creswell & Clark, 2017). In convergent designs, the purpose of the display is to present the convergent and divergent findings side by side. While none of the articles in our samples included joint displays, several articles included multiple primary and advanced elements and had high levels of mixed methods rigor. We will now discuss two such articles that employed a high level of mixed methods rigor. For each study, we will begin with a summary of the study’s context before discussing the methodology.
Exemplar Study I: It’s Not Easy Being Green: The Role of Self-Evaluations in Explaining Support of Environmental Issues
In this first study, Sonenshein et al. (2014) examine the role of self-evaluations in influencing support for environmental issues. Research suggests that supporting a social issue, whether it be climate change, gender/racial equality, corporate responsibility, or other issues, can come with a cost to the individual’s career, personal endeavors, and family commitments (Ashford et al., 1998). However, management articles do not adequately address the multifaceted context in which social issue supporters operate, both inside and outside of organizations. This study offers a richer view of both how contexts shape social issue support and how individuals’ self-evaluations play a meaningful role in understanding the experiences and, ultimately, the issue-supportive behaviors of individuals working on social issues.
The authors employed an exploratory sequential mixed methods research design. They first used qualitative data to develop theory about how environmental issue supporters evaluate themselves. They then quantitatively validated key constructs from that theory. They began the research with a focus on understanding how issue supporters interpreted and framed social issues. As is often the case in qualitative research, the authors mention, “Our research shifted as we collected and analyzed data . . . and we adjusted our focus to self-evaluations, posing the following emergent research question: How do issues supporters’ everyday experience influence their self-evaluations?” (Sonenshein et al., 2014, p. 9). In answering this question, the authors developed a grounded theory of a process that featured two core constructs that emerged from the data—self-assets and self-doubt. They describe their next steps as follows: After unpacking issue support challenges, self-assets, and self-doubts using grounded theory, we conducted a quantitative study using observational methods to inductively examine a second research question. . . . This allowed for the presentation of two different but complementary studies using mixed methods (e.g., (Creswell & Clark, 2011)) to examine both how social issue supporters’ everyday experiences influence their self-evaluations (Study 1) and why these self-evaluations matter through their ability to predict issue-related actions (Study 2). (p. 9)
The passage above highlights the author’s recognition of an advanced element of rigor (i.e., citing mixed method literature). The authors also demonstrate an understanding of rigorous qualitative analysis in the methodological discussion of the first study. In the first study, 29 interviews and two field observations were conducted and the qualitative data analysis followed a three-step process that included (1) initial data coding, (2) developing theoretical categories, and (3) theory induction (Strauss & Corbin, 1990). Furthermore, the authors offered a graphical depiction of their data structure that included first-order categories, second-order themes, and aggregate dimensions. In the second study, the authors created and validated items with a pretest that conceptualized and measured the emergent, or latent, constructs from Study 1 (i.e., self-doubts and self-assets). They then collected more quantitative data via administered surveys and conducting confirmatory factor analysis on all independent measures. They also conducted a cluster analysis to examine how participants grouped into combinations of mixed selves based on degree of self-assets and self-doubts, effectively combining rigorous quantitative analysis with the rigorous qualitative analysis.
The authors linked the two studies by stating that “study 1 showed that informants interpret being an issue supporter as posing severe challenges. These challenges have important implications for how social issue supporters stay motivated to continue to support an issue” (Sonenshein et al., 2014, p. 29). The authors discussed how Study 2 investigates these challenges. Linking the two data strands is part of the data integration process. The authors also discuss the value of the mixed methods design to begin their general discussion section: Using two complementary studies, we developed a theory of situated self-work and then examined how two core constructs in this theory—self-assets and self-doubts—related to real issue-supportive behaviors. Collectively, these two studies present a view that is an alternative to that of organizational scholars who emphasize an outward-looking self-concerned with image and reputational risk developed from contextual sensemaking inside organizations, as well as an alternative to psychological perspectives that emphasize theories that downplay context (p. 30).
This article also specifically discussed the strengths of mixed methods for their study (i.e., sampling from different field bases and presenting evidence of validated measures of emergent inductive constructs). From a best practices perspective, this article highlights the primary elements of Rigorous Mixed Methods as well as aims and purpose elements and elements of writing.
Exemplar Study II: Does Complexity Deter Customer-Focus?
In this study, Ethiraj et al. (2012) identify a theoretical tension between economic models that suggest that firms use a simple cost–benefit calculation to evaluate customer requests for new product features and the extensive management literature that shows the decision to implement innovation as being more nuanced. Specifically, this article examines how firms prioritize customer requests for incremental innovation and how product complexity creates organizational constraints that alter firm’s incentive to be customer focused.
The authors employed an explanatory sequential mixed methods research design (i.e., quantitative procedures followed by qualitative ones) to test theory-based hypotheses related to customer demand for product innovation changes and the likeliness that incremental innovations will occur at the firm level when the product is complex. First, they tested their hypotheses using a panel data set of incremental innovation decisions at a single firm. Each time the firm considered investing in an innovation requested by a customer, they collected data about the decision-making process related to the decision to innovate (or not) in response to the customer request. In all, the authors gathered and analyzed 120 customer requests, requiring incremental innovation. They performed a statistical analysis, estimating a binary choice model (using probit equations) of the likelihood of customer request fulfillment, including three sets of predictors—demand, complexity, and other controls—to account for alternative explanations. They conclude the quantitative study with the following summary: The large sample empirical analyses suggest that in the case of software products, complexity is indeed an important driver of the decision to invest resources in incremental innovation. We find that whereas customer demand is an important predictor of the decision to standardize a customer request, it has little predictive power in the decision to fulfill the request. The important unanswered question at the conclusion of the empirical analysis is what accounts for the observed results. What kinds of managerial and/or organizational decision processes explain these empirical patterns? The qualitative study in the following sections examines this question. (Ethiraj et al., 2012, p. 153)
It is in the quotation above that the authors provide a rationale for conducting mixed methods research, utilizing an explanatory sequential design. The authors also provide a mixed methods research question that asks what kinds of processes (understood through manager’s lived experience) explain the quantitative results.
The qualitative study involved in-depth interviews with heads of product development, product management department, and marketing and sales staff. The focus of the qualitative study was to understand the innovative decision-making process and how it might account for the observed quantitative results. In sum, the qualitative study identified three underlying elements of the decision-making process (or themes) that might prevent the demand driven innovation (DDI) model from explaining investment in incremental innovation—the importance of organizational structures, competitive pressures, and incentives for resource allocation processes. Data integration occurred in the discussion section that summarizes the theoretical contributions of the study. The quantitative results identified patterns related to whether or not to allocate resources for product innovations and how firms implemented the investment. The qualitative study revealed that fulfillment decisions were dependent on the firm’s decision-making processes related to resource allocation. While the previous exemplar did a good job of connecting the data between the qualitative and quantitative studies, this article demonstrates the inherent value of mixing both data types. Overall, the authors were able to make stronger conclusions by using the strengths of each monomethod design type. This study represents an explanatory sequential design, one that highlights the primary elements of Rigorous Mixed Methods as well as provides a rationale for the use of mixed methods.
These two exemplar articles demonstrate that Rigorous Mixed Methods research is being done in management studies, but the overall data set suggests that the discipline as a whole could benefit from the Rigorous Mixed Methods criteria previously presented. Each piece presented here had strengths that merited its inclusion in this study and allow for prospective authors to model their work after existing, successful, publications. The following discussion highlights the current state of mixed methods in management research and discusses ways in which the field as a whole may move forward.
Discussion
Contribution to the Field of Mixed Methods Research Methodology
This article contributes to the field of mixed methods research by creating a framework for conducting and reporting mixed methods studies in a rigorous manner. This varies from previous work on conducting “high-quality” or “good” mixed methods as any debate about quality should be viewed through the lens of how a researcher was trained in their academic discipline. Researchers can disagree about what constitutes a high-quality article, yet this framework aims to provide guidance for recognizing what qualifies as highly rigorous mixed methods. The discussion of Rigorous Mixed Methods procedures is a relatively new phenomenon and as such has been mostly limited to the evaluation, education, and health science literatures. This article advances this ongoing discussion by providing more evidence from a business discipline, allowing for the discussion of discipline-specific influences that can affect the development of a research method over time. One example of this is the relatively strong focus on quantitative methods in the business literature (i.e., exploratory sequential designs with a quantitative emphasis were most common in the data). This may be due to a general quantitative emphasis in the business disciplines, as well as publishing trends within the journals reviewed. Further work is needed to identify the “paradigmatic” lens with which mixed methods is viewed within a given discipline, which may at some time allow for an overall discussion of mixed methods quality, but that cannot happen until a common agreement on rigorous methods is achieved.
Our findings also highlight patterns in Rigorous Mixed Methods across different journal outlets. For example, while the Academy of Management Journal and the Strategic Management Journal both have approximately the same number of articles that combine qualitative and quantitative data, the Academy of Management Journal had a higher percentage of rigorous articles. The Academy of Management Journal is considered one of the highest ranking management journals; thus, perhaps the most rigorous articles in the discipline are more likely to be published in this journal. While the Journal of Management included more articles, the journal did not contain a study that rated medium–high or higher in Rigorous Mixed Methods. This journal is relatively new and may be more rigid in its orientation to alternative methods. Administrative Science Quarterly only had one study with such a rating. So how are readers to interpret these findings about journals that offer limited examples of redundant Rigorous Mixed Methods? Are they less oriented toward or less knowledgeable about the tenets of the approach? Or, are there more systemic, political, or historical reasons for the results? Bazeley’s (2009) review of management literature concluded that there is continuing prominence of quantitatively based, statistical approaches in management research. Furthermore, Hurmerinta-Peltomaki and Nummela (2006) suggest that mixed methods research faces publication challenges due to paradigmatic views within the management discipline. Cameron (2011) suggests that some management journals exclude mixed methods research, explicitly and implicitly through methodological preferences. Other issues include page restrictions that limit how exhaustive an article can be in terms of mixed methods rigor and still fit within the number of pages allocated by journal editors. While we argue strongly for the usefulness and feasibility of mixed methods research within a journal’s page constraints, we do acknowledge this as a weakness of the method. With that said, one way to include methodological detail, in light of page restrictions, could be to use appendices or online open access forums to present additional material.
Overall, our findings reveal that Rigorous Mixed Methods articles are published in the management literature. In these articles, the authors collected and analyzed qualitative and quantitative data, showed some integration of the two data strands, and often indicated that they were thinking about the rationale for why they conducted mixed methods research. Furthermore, the core components of Rigorous Mixed Methods were well represented in the high-ranking articles. However, while our findings offer several best practice examples, a majority of the articles found in this analysis would not rate as highly rigorous according to the mixed methods literature and the Rigorous Mixed Methods framework. Overall, 65.6% of articles reviewed for this article qualified as having low mixed methods rigor. Again, these articles are not “true” mixed methods studies by many definitions. This suggests significant room for improvement in the utilization of mixed methods in the management discipline. Of the 195 articles, only 9.7% were deemed to have medium to high methodological rigor. This shows that while highly Rigorous Mixed Methods work is being done, it has yet to become the norm.
Furthermore, of the articles in this analysis, only one referenced the extant mixed methods literature, and only two were labeled as mixed methods articles. None of the articles identified a specific mixed methods design type or included a joint display of the findings. Referencing mixed methods literature and labeling articles as mixed methods give a study more sophistication and signal an understanding of the distinct methodological approach. Tapping into the existing body of mixed methods literature could lead to more rigorous articles, as authors become aware of the challenges and benefits of established approaches. Citing relevant mixed methods literature also helps editors identify manuscript reviewers. Utilizing a specific research design opens up the possibility of knowing where the challenge points exist in the different mixed methods approaches. Identifying the design type also affects the procedures that unfold, as mixed methods scholars often match the writing structure, the mixed methods research question, the title, and the specific design type, because design types become central to tying together various structural elements (Creswell & Clark, 2017). The presentation of joint displays and visual methodological diagrams is a selling point for stakeholders, research teams, and journal reviewers because it facilitates an understanding of the project, particularly important in the practitioner field like management. Overall, these issues show the value and need for the Rigorous Mixed Methods framework proposed in this article.
So how can authors enhance the level of mixed methods rigor in the field of management studies? Clearly, this can be done by improving empirical projects by adding in Rigorous Mixed Methods elements that were missing in some of the articles. As the mixed methods literature develops in management studies, it is likely that the discipline will begin to see more systematic initiatives to enhance mixed methods research rigor and, in turn, quality as appreciated by management scholars. This may include emboldening journal editors to open up to mixed methods, as it appears that some journals are much more receptive than others. Academic conferences, panels, and workshops on mixed methods may also foster the debate necessary to introduce or change thoughts about what are rigorous research methods. Other initiatives or opportunities that can potentially ignite institutional change include the development of books tailoring mixed methods to the management context. What makes specific mixed methods design types particularly useful for management scholars? Several disciplines have developed texts that address contextualized issues related to both methodological rigor and design, adapted specifically for scholars in their discipline. Management studies have the opportunity to follow suit. Regardless of the approach, it is not a revolutionary change that will improve the rigorousness of management studies—but an evolutionary one. As mixed methods researchers continue to discuss issues related to improving the use of the method, we have the opportunity to change mind-sets related to the value and rigorousness of the approach, while also providing guidance for editors, reviewers, and researchers.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
