Integrating Literature as a Data Source in Mixed Methods Research

Introduction

Mixed methods research integrates qualitative and quantitative data and is recognized as the third research paradigm (Creswell & Plano Clark, 2018; Johnson et al., 2007). Utilizing mixed methods often provides more ‘…informative, complete, balanced, and useful research results’ (Johnson et al., 2007, p. 129) compared to quantitative or qualitative methods used in isolation. The application of mixed methods research fosters the understanding of complex issues which is essential for solving real world problems (Molina-Azorin & Fetters, 2019). Mixed methods research has become increasingly popular in recent years, especially in social and behavioral sciences (Timans et al., 2019) and, like all areas of research, is continually evolving (Creswell & Garrett, 2008; Timans et al., 2019).

Integration, which is an essential component in mixed methods research, is a process where researchers intentionally draw together quantitative and qualitative approaches (Creswell, 2015; Guetterman et al., 2015; Younas et al., 2020). Despite being a hallmark of mixed methods research, integration is not clearly developed or practiced and when it occurs, integration varies based on the design used (Guetterman et al., 2015; Halcomb & Hickman, 2015; Ivankova et al., 2006; O’Cathain et al., 2010). While improvements have been made, with new publications to assist researchers to understand and practice integration, issues with suboptimal integration continue to be frequently observed in published empirical mixed methods studies (Fetters & Tajima, 2022; Granikov et al., 2020; Irvine et al., 2019). Fetters and Freshwater (2015) discuss the need to address the ‘integration challenge’ in order to maximize the benefits of integration, that is to achieve a sum greater than the individual qualitative and quantitative parts. Mixed methods researchers create studies that can include integration at the design, methods and interpretation and reporting levels (Fetters et al., 2013).

To characterize how integration occurs, Leech and Onwuegbuzie (2009) suggested three key considerations for mixed methods study designs; the level of mixing, time orientation and the emphasis of approaches. These considerations encompass whether the methods are fully or partially mixed, whether methods are used concurrently or sequentially and whether the quantitative or qualitative approaches have equal versus dominant status (Leech & Onwuegbuzie, 2009). In a literature review of mixed methods design typologies Nastasi et al. (2010) embedded the concepts described by Leech and Onwuegbuzie (2009) into a range of criteria used to classify studies, and summarized the existing typologies as Types I through to VI, distinguishing between basic and complex typologies.

At the methods level, four main systematic integrative techniques are described in the literature: connecting, building, merging and embedding (Creswell & Plano Clark, 2018; Fetters et al., 2013). Connecting is when one type of data links with the other through the sampling frame, building is when one data collection procedure informs the other data collection approach used, merging refers to bringing together and comparing quantitative and qualitative data and embedding involves linking data collection and analysis at multiple points (Fetters et al., 2013). There is recognition that there is often uncertainty about how to integrate the data collected for mixed methods studies (Bryman, 2007; Fetters, 2020). This uncertainty may result in data that are not integrated or limited integration of data (Bryman, 2007).

In order to address this uncertainty, Creswell and Plano Clark (2018) proposed that four key elements are required to achieve integration: (i) the rationale for using data integration, (ii) a clear explanation of the procedures for data analysis and integration, (iii) a description of how the integrated findings will be represented and (iv) interpretations of the integrated results. Younas et al. (2020) suggest that the first two elements (i, ii) can be addressed through narrative discussion in the methods sections and joint display development, and analysis can be utilized to address presentation (iii) and interpretation of integrated results (iv). At the interpretation and reporting level, Fetters et al. (2013, p. 2142) outline three approaches for the integration of data as ‘(1) integrating through narrative; (2) integrating through data transformation; and (3) integrating through joint displays’. This paper focuses on the integration of findings from the literature with quantitative and qualitative data into joint displays.

Example 1: The Impact of Organizational Values on Nurse Resilience (Cooper, 2021)

Integration Process

This study utilized integration at the method, interpretation and reporting level, guided by Fetters et al., (2013). Integration occurred at the method level with integration through connecting. Quantitative data were collected via cross-sectional survey in the first phase and qualitative data were collected via focus groups in the second phase. Focus group participants were selected from respondents to the questionnaire. In keeping with the concurrent design, most integration took place after data collection at the interpretation and reporting level, ​where findings from literature sources, and QUAN, and QUAL results were drawn together (merging). ​Interpretation and comparison across the four data sources were completed using a joint display.

Displaying and comparing the findings from each data source in a joint display table assisted in identifying areas of confirmation, expansion and discordance between the data sources (Fetters et al., 2013). Key related findings and results from each data source, where applicable, were summarized in the joint display table and color coding was used to visually highlight confirmation (in white), discordance (in orange) or expansion (in blue) across the data sources. An excerpt of the joint display (Table 1), presents findings from the CA and IR, alongside results from the quantitative and qualitative data.

OPEN IN VIEWER

Table 1.

Cross-Case Analysis Matrix.

Example 2: Understanding and Addressing Type 1 Diabetes-Specific Barriers to Physical Activity (Brennan, 2021)

Integration Process

Integration occurred at three levels: design, methods and interpretation and reporting (Fetters et al., 2013). Design integration emphasized the quantitative phase of this study over the qualitative phase (QUAN → qual) (Ivankova et al., 2006; Tashakkori & Teddlie, 2003). Qualitative inquiry was used to expand on and provide explanations of the quantitative findings (Fetters et al., 2013). Integration through methods occurred in several ways: embedding, connecting and building (Creswell & Plano Clark, 2018; Fetters et al., 2013). The qualitative data were embedded as a secondary, complementary dataset within the larger study to understand contextual factors that influenced quantitative findings and to understand participants’ experience of the intervention and study procedures. Quantitative data connected with qualitative data through the sampling frame; participants who did not withdraw from the RCT were invited to attend a focus group interview. Finally, specific quantitative results that required additional explanation were identified and used to develop the focus group interview questions (building) (Creswell, 2009; Creswell & Plano Clark, 2018; Fetters et al., 2013). Integration of quantitative and qualitative data at the interpretation and reporting phase occurred using a joint display (Table 2) (Creswell & Plano Clark, 2018; Fetters et al., 2013).

OPEN IN VIEWER

Table 2.

Cross-Case Analysis Matrix.

Quantitative and qualitative results were initially analyzed and published separately (Brennan et al., 2021, 2021c). Data were then brought together using a joint display table (partially represented in Table 2). Table 2 first shows how key findings of the systematic scoping review informed the mixed methods study objectives. Quantitative and qualitative results are then presented alongside whole-of-study outcomes and implications. After presenting quantitative and qualitative results side-by-side, the fit of data integration was determined, that is, how the qualitative results confirmed, expanded or in the case of this explanatory sequential design, explained, and or were discordant to quantitative findings (Creswell & Plano Clark, 2018; Fetters et al., 2013). A discussion of how these meta-inferences relate and compare to the original systematic scoping review findings was provided in narrative form (Brennan, 2021).

Interpretation of the Joint Display

As depicted in Table 2, the systematic scoping review called for patient education on blood glucose management and to address diabetes-specific barriers to physical activity in the T1D population. This finding informed mixed methods research objective 2: to determine the acceptability of the intervention and control. Meta-inferences, represented in green, show positive intervention fidelity outcomes were confirmed by qualitative data (see Table 2). Qualitative data expanded on quantitative data pertaining to improved participant retention and infrequent Facebook^™ group use, described in orange in Table 2. Finally, qualitative findings expanded (orange) on the quantitative fidelity findings, indicating the control may have provided more than ‘standard care’. This comprehensive mixed methods assessment of intervention acceptability has shown that theory-driven, self-management, group education to address complex diabetes-specific barriers to physical activity was acceptable to adults living with T1D, addressing an important gap identified in the systematic scoping review (Brennan et al., 2021c), displayed in the first column of Table 2.

Discussion

We build on the call from Fetters and Freshwater (2015) to address the ‘integration challenge’ by demonstrating the process and value of integrating literature sources in mixed methods research. The two examples presented in this paper showcase how integration can be achieved in different study designs. Integration of literature sources in mixed methods research is novel and represents an evolution in mixed methods design, requiring a revision of the typologies currently used to describe the level of mixing, time orientation and the emphasis of approaches (Leech & Onwuegbuzie, 2009).

The first example illustrated how data from literature sources (LIT) were synthesized to address study objectives, and could be integrated with QUAN + QUAL data. To clearly articulate the design, this study is best described as a multi-stage, partially mixed, time-lapsed concurrent, equal status mixed methods design, with the new notation expressed as LIT + QUAN + QUAL. This reflects the equal status of each data source and how each data source contributed to addressing the overall study objectives. The second example showed how a literature source was used to inform the mixed methods study objectives and could then be integrated with QUAN → qual data. This design is best described as a multi-stage, explanatory, sequential, mixed methods design, with the new notation lit →QUAN → qual. This reflects the sequential nature of the study with the literature source informing the study objectives and preceding the QUAN and qual phases.

At the methods level researchers need to consider what systematic integrative techniques to use in their study. Connecting, where one type of data links with the other through the sampling frame (Creswell & Plano Clark, 2018; Fetters et al., 2013), is unlikely to apply to the integration of literature sources, but may still exist between the quantitative and qualitative study components. The other three main systematic integrative techniques, building, merging and embedding (Creswell & Plano Clark, 2018; Fetters et al., 2013), can all be used when integrating literature sources. Building was evident in the second example, with data synthesized in the scoping review informing the study design and data collection approach used in the QUAN and qual stages of the study. Both examples demonstrated merging, where literature, quantitative and qualitative data sources were brought together and compared. Embedding occurred in the first example, where qualitative data and literature helped to explain and give context to quantitative results, revealing the contextual factors that could influence nurse resilience and the perceptions and experiences of participants.

For the interpreting and reporting level, we recommend joint display as an effective technique to assist researchers in the side-by-side comparison of literature, quantitative and qualitative data sources. Side-by-side comparison using joint displays enhances the mixed data analysis process and provides a clear representation of the ensuing inferences (Creamer & Edwards, 2022; Fetters & Tajima, 2022; Younas et al., 2020). We describe two examples where this method has allowed comparison of three data sources, to begin an iterative process of determining the ‘fit’ and meta-inferences of the data through confirmation, expansion and discordance (Brennan, 2021; Cooper, 2021; Fetters et al., 2013; Younas et al., 2020). Building on the four best practices for developing a joint display outlined by Guetterman et al. (2015) we recommend an additional best practice when integrating literature sources using joint display is to label the literature source findings with the inclusion of a dedicated column for these findings. This results in five best practices, namely, (i) label quantitative and qualitative results, (ii) label literature source findings , (iii) be consistent with the design, (iv) be consistent with the integration approach and (v) identify inferences or insights generated.

Given the vast amount of data to collate in both examples, joint displays were an effective and efficient method to organize and present a cohesive and systematic comparison of multiple data sources beyond traditional quantitative and qualitative sources. Although the use of joint displays are common, authors are not making adequate use of joint displays as a tool to integrate quantitative and qualitative findings, nor to integrate beyond these traditional data sources (Younas et al., 2020). The examples presented in this paper were from two large projects, submitted individually for the awards of Doctor of Philosophy (Brennan, 2021; Cooper, 2021). The projects required a clear and consistent method to enable the integration and discussion of multiple data sources beyond traditional mixed methods findings.

In presenting these examples we argue that literature reviews and concept analyses are valid sources of data, warranting integration with quantitative and qualitative data, to create a sum greater than quantitative and qualitative findings alone. We promote this method of integration to researchers who seek to link literature reviews to their own original mixed methods data and to communicate complex research conclusions with relative ease.

Effective and efficient communication of research findings is crucial for successful research translation (Straus et al., 2013). Often, the reader must navigate a lengthy narrative discussion to understand how the findings of a mixed method study relate to existing literature and fills knowledge gaps. Presenting and integrating multiple data sources, including findings from the literature, via joint display, positions the reader to think simultaneously about all sources of data, and how they relate to each other (Haynes-Brown & Fetters, 2021), while allowing the reader to appreciate the intent and technique of integration (Guetterman et al., 2015; Younas et al., 2020). Furthermore, presenting data sources in this way is likely to improve the readability and interpretation of the ensuing narrative discussion. The joint displays presented in this article illustrate effective and efficient communication of mixed methods results, while enabling the reader to situate this new knowledge among gaps found in the extant literature (Brennan, 2021; Cooper, 2021).

Systematic (mixed) literature reviews and concept analyses were presented as additional data sources in the examples, but are not an exhaustive list of alternative data sources suitable for integration. Establishing a sound research question to address gaps identified in the wider literature is the foundation of any robust study design (Creswell & Plano Clark, 2018; Moher et al., 2010; World Medical Association, 2013). As such, integrating literature sources has wide-reaching value to all researchers who aim to link the extant literature to their research objectives and research findings. Further, consumers of research and those translating research findings into practice, can be assured that new evidence recommending practice change, has considered the existing concepts and context. We demonstrate through the included examples how joint displays can weave a research ‘story’ together and complete the circle of knowledge, providing a visual representation of the significance of the study in relation to the identified gaps in the literature.

As these two examples illustrate, the integration of literature sources will generate new mixed methods designs that will require new typologies and notation to convey accurate descriptions. Researchers need to carefully consider the purpose of their literature source(s), and as in other designs, describe the level of mixing that will occur, the time orientation and emphasis of approaches (Leech & Onwuegbuzie, 2009). We encourage research students and active researchers to apply our approach to literature integration to their own work, and to explore opportunities to include other, non-traditional sources of data into joint displays. In doing so, further methodological advancements will become apparent, further advancing the field of mixed methods research.

Contribution to the Field of Mixed Methods Research Methodology

Joint displays have emerged as an innovative method to integrate and interpret mixed methods data and have been used across various mixed methods designs to generate new inferences (Guetterman et al., 2015). Over the past decade, mixed methods researchers have begun to use joint displays in more innovative ways, for example, integrating mixed methods results using a theoretical framework (Bustamante, 2017), using the pillar integration process (Johnson et al., 2019), incorporating graphical displays (Haynes-Brown & Fetters, 2021) and using meta-displays that integrate findings from multiple joint displays (Creamer & Edwards, 2022; Fetters & Guetterman, 2021; Peters et al., 2021). We have detailed and expanded upon a method to enrich the integration process by incorporating literature sources for integration using joint display.

While researchers are urged to demonstrate synergy between traditional quantitative and qualitative data sources (Fetters & Freshwater, 2015; Guetterman et al., 2015), there is no guidance, nor examples of integrated mixed methods meta-inferences with literature sources. Aligning these meta-inferences with results from literature reviews and concept analyses using joint displays delivers rich and concise communication of complex research problems. This approach builds on the concept of maximizing the benefit of integration discussed by Fetters and Freshwater (2015) by offering a method to display findings that are greater than the sum of the individual components.

Limitations and Future Research

We acknowledge our discussion is limited to two examples of integrating literature sources using joint display, both in the healthcare field. The use of different mixed methods research designs and literature sources to address real world problems does, however, strengthen this discussion, by highlighting the adaptability of this method. The two examples do not represent all the ways that literature sources can be used in mixed methods research designs. The exemplar joint displays presented above do not clearly demonstrate all the levels at which integration can occur, partly because joint displays are typically used to integrate at the analytic and interpretation level (Guetterman et al., 2015). This may be a limitation of joint displays more broadly, but when coupled with a cohesive narrative, other areas of integration become clear. There are many other ways joint displays can be utilized that are beyond the scope of this discussion; however, the examples presented will allow researchers to consider if and how including literature sources may strengthen integration. Future applications and evaluation of literature sources using joint display in different mixed methods research designs, using different data sources and in disciplines outside of healthcare, are required to advance this method of integration.

Conclusion

We present a novel method of integrating literature sources using joint display that advances the utility of joint display and builds on the concept of creating a sum that is greater than quantitative and qualitative findings alone. The two examples presented address real world problems of sustaining nurse resilience and barriers to and facilitators of physical activity in adults living with T1D. Joint display is an established method of integration at the analysis and interpretation level, that continues to evolve. To maximize the benefits of integration, researchers have begun to use joint display in creative and innovative ways (Bustamante, 2017; Guetterman et al., 2015; Haynes-Brown & Fetters, 2021; Johnson et al., 2019). The inclusion of literature sources at the interpretation and reporting level facilitates a deeper understanding of research phenomenon and assists researchers to address societal problems (Molina-Azorin & Fetters, 2019). Literature sources can also be used to inform mixed methods research designs and address wider study objectives.