Abstract
Findings from knowledge-building and theory-generating qualitative systematic reviews have the potential to help guide policy formation and practice in many disciplines. Unfortunately, this potential is currently hindered by the fact that rigorous data analysis methods have not been consistently used and/or articulated for purposes of conducting these types of reviews. Content analysis is a flexible data analysis method that can be used to conduct qualitative systematic reviews; however, its application in this context has not been fully explicated. Qualitative systematic reviewers who aim to build knowledge and generate theory are urged to adapt content analysis methods to accommodate data that are, by nature, highly organized and contextualized. In addition, they are encouraged to use reflective memoing and diagramming to ensure valid integration, interpretation, and synthesis of findings across studies. Finally, reviewers are advised to clearly and fully explain their data analysis methods in research reports.
Keywords
In response to increasing calls for evidence-based practice, researchers have sought new ways to respond to questions that are difficult to answer using quantitative methods. Often, these questions pertain to complex human phenomena that may be most easily answered using qualitative methods. That said, scholars have found it challenging to move knowledge development and theory generation forward based on findings from isolated qualitative research investigations (Leeman and Sandelowski, 2012). To overcome this problem, researchers (e.g. Paterson et al., 2001; Sandelowski and Barroso, 2007) have developed methods to analyze qualitative findings from across multiple investigations through systematic review (Campbell et al., 2011; Finfgeld-Connett, 2010b).
Despite numerous efforts to develop and articulate rigorous qualitative systematic review methods (e.g. Paterson et al., 2001; Sandelowski and Barroso, 2007), few attempts have been made to specifically and precisely describe data analysis processes (Campbell et al., 2011; Hannes and Macaitis, 2012). In their seminal work relating to qualitative systematic review, Noblit and Hare (1988) suggested that since the research genre is new, data analysis strategies will emerge over time. Since the late 1980s, one method that has been adapted for this purpose is content analysis (e.g. Holm and Severinsson, 2012; Rissanen et al., 2011).
Qualitative content analysis is a flexible data analysis method that can range from impressionistic interpretations to highly systematic analyses of text-based data (Hsieh and Shannon, 2005). It is considered a qualitative method for systematically and rigorously integrating, interpreting, and synthesizing qualitative findings that have been extracted from multiple qualitative or mixed-method research reports. The author’s aim in this article is to make this data analysis method more transparent for purposes of conducting knowledge-building and theory-generating qualitative systematic reviews.
Qualitative content analysis
A hallmark of qualitative content analysis is coding raw data into conceptually congruent categories (Elo and Kyngas, 2008; Hsieh and Shannon, 2005). In the case of qualitative systematic reviews, raw data consist of qualitative research findings (i.e. text) that have been systematically extracted from existing research reports (Finfgeld-Connett, 2010b). The manner in which these findings are coded is largely guided by the research topic and questions and the data that are available for analysis.
Content analysis is also influenced by whether an inductive or deductive approach is used (Elo and Kyngas, 2008; Hsieh and Shannon, 2005). In the case of inductive analysis, qualitative systematic reviewers begin a review with few preconceptions about a topic, and they do not have a coding framework in mind. Instead, they start the analysis process by studying the raw data (i.e. qualitative findings) and making evidence-based inferences about organizing codes.
Using a deductive approach, the reviewer begins data analysis with a coding template in mind, and data are organized according to an existing, though alterable, structure. Alterability is important since one aim of qualitative systematic reviews is to test, adapt, expand, and in general, improve upon the relevance and validity of existing frameworks (Finfgeld, 2003; Zimmer, 2006). Guiding frameworks may be quite broad, such as those that are associated with grounded process theory (e.g. Corbin and Strauss, 2008), or they might be quite specific. The latter is more common when reviewers have already examined a topic in depth or when they have personally conducted primary investigations relating to the topic of interest (Elo and Kyngas, 2008; Hsieh and Shannon, 2005).
Regardless of whether content analysis is approached inductively or deductively, threats to validity exist. Implicit in the deductive approach is the tendency to verify the obvious and to overlook opportunities to expand or refute theoretical tenets (Hsieh and Shannon, 2005). Conversely, the inductive approach is associated with the tendency to overlook the obvious and perpetually reinvent the wheel, which can result in failure to build knowledge (Campbell et al., 2011). Strategies (e.g. memoing, diagramming, and reflection) for diminishing these threats to validity are discussed later.
Process of qualitative content analysis
The process of qualitative content analysis consists of several steps. For clarity, these steps are discussed as if they are discrete and unidirectional. In reality, they may be revisited and repeated many times as emergent insights dictate. This may mean until the final research report is complete (Campbell et al., 2011).
Identification of data segments
To gain a sense of the topic as a whole, qualitative systematic reviewers begin the content analysis process by carefully and reflectively reading multiple research reports from the sample database. When using an inductive approach, this process of assiduously reading and reflecting results in the tentative identification of organizing codes, which are initially noted in the margins of each research report (Elo and Kyngas, 2008; Hsieh and Shannon, 2005).
During this early stage of document review, it is recommended that careful consideration be given to the size of the data segments (e.g. sentence fragments, sentences, paragraphs, etc.) (Elo and Kyngas, 2008; Graneheim and Lundman, 2004). This is especially important when conducting qualitative systematic reviews since the research findings that are used for this purpose are already highly contextualized. Thus, the threat is that fine-grained coding (e.g. open coding (Corbin and Strauss, 2008)) of small data segments will result in meaningless (i.e. deconstructed) findings (Finfgeld, 2003). Conversely, when dealing with excessively large data segments, the threat is that the text will be coded too abstractly to be fully meaningful (Miles and Huberman, 1994). In both instances, the result is inscrutable findings that cannot be easily applied for purposes of policy formation or practice.
The purpose and questions behind a qualitative systematic review will provide some guidance regarding the most advantageous data segment size and level of coding (Elo and Kyngas, 2008). For example, while examining the context and helping strategies that are used to assist domestically abused South Asian immigrant women, fine-grained coding proved to be counterproductive, since too much contextual information was stripped and emergent findings were rendered meaningless (Finfgeld-Connett and Johnson, 2012b). Conversely, in an attempt to identify new ways of explaining homelessness among women with substance abuse problems, more nuanced coding was necessary to see beyond conventional paradigms and to articulate new explanations (Finfgeld-Connett et al., 2012a).
Data matrices and coding
As data segments and accompanying codes are tentatively identified, electronic matrices (i.e. tables) should be developed for the receipt of coded data from each report. To maintain a clear audit trail, it is recommended that reference citations be placed in the far left column. Subsequent columns should be labeled using preliminarily identified codes, and extracted raw data (i.e. qualitative findings in the form of data segments) are then placed into these columns (Averill, 2002; Miles and Huberman, 1994) (see Table 1).
Example of coded findings and initial memos. a
From qualitative systematic review relating domestic abuse among South Asian immigrant women (Author A and others, 2012b).
While processing data from individual study reports, qualitative systematic reviewers are urged to use caution when combining codes to form more abstract categories. As noted earlier, qualitative findings that constitute the raw data for systematic reviews already represent highly processed and contextualized research findings versus fragmented ideas stemming from impromptu responses to research questions. At a minimum, overprocessing of qualitative research findings can result in needless work, and at worst, misrepresentation of the data (Sandelowski et al., 2008).
Following the careful organization of qualitative findings into matrices, it is necessary to integrate, interpret, and synthesize these data and their meanings across study reports. Inevitably, research studies that comprise the sample for a qualitative systematic review will be topically, paradigmatically, and methodologically similar, but at the same time different. Consequently, it follows that qualitative findings that are extracted from each research report will be similar but different, and meaningful interpretations across studies may take shape slowly. It is recommended that these interpretations be developed through a process of memoing, diagramming, and reflection.
Memoing
Reflective memos are written notations about the data that are drafted as data analysis progresses. Throughout this process, coded findings are organized, clarified, integrated, and interpreted (Corbin and Strauss, 2008; Lempert, 2007; Miles and Huberman, 1994). The intended outcome is synthesis of findings across studies and inferences for policy formation and practice.
In the interest of maintaining a precise audit trail and remaining grounded in the research findings (Birks et al., 2008), analysts are urged to initially place memos adjacent to the raw data. To accommodate this, memo columns are inserted into coding matrices so that memoranda and raw data can be positioned side by side (see Table 1).
Memoing begins by reflectively studying coded qualitative research findings and organizing and clustering these data so that the findings are made clear and transparent. As necessary, findings are carefully paraphrased or reworded to clarify meaning and to make the data fully available for analysis (Graneheim and Lundman, 2004; Sandelowski et al., 2008). Depending on the number of research reports in a sample, development of initial within-study memos can be a tedious and time-intensive process, but in the end, the researcher is left with a keen awareness of the raw data and the potential to integrate, interpret, and synthesize ideas across studies.
Once within-study memoing is complete, the task of integrating, interpreting, and synthesizing memos across studies ensues. To begin this process, similarly coded memos should be grouped together so that they can be easily reviewed and analyzed in their entirety. Based on ongoing memoing, opportunities to combine and collapse codes tend to become evident. In addition, interconnections between and among codes and categories are likely to emerge (see Table 2).
Example of formative memos across studies. a
From qualitative systematic review relating domestic abuse among South Asian immigrant women (Finfgeld-Connett and Johnson, 2012b).
Disconnected parallel ideas, no matter how saturated or well explicated, are not ideally suited for use in policy formation and practice. Findings from qualitative systematic reviews will garner the greatest utility if they interconnect to form a whole. Interconnections may consist of shared attributes of a single phenomenon, antecedents or consequences of a process, or elements that intersect a system (Morse and Singleton, 2001). Through iterative memoing, the researcher is able to examine these hypothetical interconnections, and depending on their degree of saturation and fit, accept or reject them.
It is noteworthy that once similarly coded and categorized memos are grouped together (as in Table 2), they are no longer situated directly beside the raw data. The obvious disadvantage of this arrangement is that the researcher can no longer directly rely on the original study findings for verification, and the veracity of the review results is increasingly dependent on the iterative process of reading, reflecting, and revising formative memos (Corbin and Strauss, 2008; Noblit and Hare, 1988). The practical advantage of this configuration is that the researcher is compelled to think more abstractly about the data, and the process of integrating, interpreting, and synthesizing formative memos across studies is enhanced.
Diagramming
In concert with memoing, systematic reviewers are urged to reflectively diagram how codes, categories, and accompanying memos appear to relate or fit together. Reflective diagramming is in keeping with the tradition of field research, wherein researchers continually observe and diagrammatically record and interpret their observations. This type of visual analysis is particularly helpful in terms of assisting researchers to identify inconsistencies and gaps in their thinking (Keller, 2011).
Diagramming does not require sophisticated software or artistic ability. On the contrary, working diagrams may largely consist of primitive line drawings, sketches, or sticky notes that are assembled on a flipchart (Keller, 2011). Their importance stems from the fact that the researcher is able to visually try out relationships between and among elements of an emergent model or phenomenon and make revisions as needed. This process also serves as an additional way of documenting data analysis decisions and the thinking behind them (Corbin and Strauss, 2008; Lempert, 2007).
Many types of relationships between and among emergent systematic review findings are possible, including contiguous, linear, circuitous, hierarchical, and temporal (Lempert, 2007). In addition, the systematic reviewer might identify codes and memos that lie outside of an emergent schema. Identification of elements that are not consistent with a formative model is entirely possible when conducting qualitative systematic reviews since, as noted earlier, research topics and questions vary within the review sample, and not all findings may apply to a single systematic review (Finfgeld-Connett and Johnson, 2011).
Moreover, it is entirely possible that a reviewer may identify qualitative research findings that lack credibility within an emergent model. This may relate to the fact that the original research involved methodological weaknesses that jeopardize the veracity of the findings (Pawson, 2006). Unfortunately, no amount or type of vetting may reveal these problems at the outset of a systematic review since instruments for determining the validity of qualitative research are imperfect, authors do not always clearly disclose methodological flaws, and reporting practices may obscure issues related to veracity (Campbell et al., 2011; Pawson, 2006). More important with regard to this discussion is the fact that through a process of rigorous coding, memoing, and diagramming, systematic reviewers have the potential to identify findings that lack credibility and to suggest directions for future qualitative studies.
Not only are diagramming and memoing essential with regard to data analysis, they are also helpful in terms of propelling the systematic review forward. For instance, when data analysis seems stalled, diagramming and memoing have the potential to inspire new insights. In addition, by diagramming and memoing, the researcher is able to see cumulative progress as findings emerge and summative results take shape (Birks et al., 2008; Corbin and Strauss, 2008).
Reflection
Although skilled reflection is imperative during all stages of the qualitative systematic review process, reflective thinking is particularly important as the systematic reviewer integrates, interprets, and synthesizes ideas across research reports. As this occurs, researchers are challenged to remain grounded in the original qualitative findings while synthesizing and articulating new insights across studies. Based on a cyclic process of reading, writing, reflecting, and revising, ideas are brought together, connections are made, and concepts and processes are gradually constructed (e.g. Finfgeld-Connett, 2010a).
More explicitly, formative ideas and their interconnections are examined through multiple lenses and tentatively tried out in hypothetical situations. As necessary, ideas are retained, held in abeyance, or abandoned altogether. Based on a process of constant iterative reflection, it is difficult to ignore personal biases and contradictory evidence, and researchers are driven toward the synthesis of evidence-based concepts and propositions (Birks et al., 2008; Jasper, 2005; Mruck and Mey, 2007). This iterative process continues until the findings are fully explicated, and the synthesized review findings resonate as true based on the contexts in which they originated and those in which they are anticipated to apply (Finfgeld-Connett et al., 2012a).
It is during the process of reflective memoing and diagramming that team members or outside consultants, such as program committee members, may offer the most help during the data analysis process. As noted earlier, qualitative research findings are, by nature, highly contextualized, and overzealous coding (i.e. deconstruction) may result in misrepresentation (Sandelowski et al., 2008). Thus, collaborators’ viewpoints and perspectives might be most helpful while reflectively memoing and diagramming across studies (e.g. Finfgeld-Connett et al., 2012a).
Data saturation and fit
When conducting knowledge-building and theory-generating qualitative systematic reviews, validity of the findings is gauged based on data saturation and fit (Finfgeld-Connett and Johnson, 2013), Saturation is achieved when the review findings are fully grounded, and additional data do not enhance understanding or meaning (Corbin and Strauss, 2008). Campbell et al. (2011) acknowledge that this is likely to occur when multiple qualitative investigations have been conducted on a topic. They and others (O’Reilly and Parker, 2012) suggest, however, that mere saturation of findings does not necessarily result in knowledge development. Instead, the criterion of fit is equally important in terms of extending a conceptualization or, alternatively, understanding what is missing (Morse and Singleton, 2001).
The criterion of fit is not based on the simple replication of isolated findings across multiple studies. Rather, fit pertains to how findings interrelate to form a systematic whole. As mentioned previously, there are several ways in which grounded findings may fit together, including contiguously, linearly, circuitously, hierarchically, and temporally. As part of a whole, elements are also expected to intersect and impact one another (Morse and Singleton, 2001). When the criterion of fit is not met, and findings remain unlinked, saturated data silos persist, and the full potential of conducting a knowledge-building or theory-generating systematic review is not realized.
Although the risk of force fitting findings exists, this threat is diminished by reflective data analysis processes, wherein false assumptions are likely to be exposed as ungrounded (Morse and Singleton, 2001). In the end, some associations may be deemed more tentative than others; however, when this occurs, new directions for research can be identified, and knowledge development is enabled versus remaining stagnant.
Epistemology
Debate exists, although diminishing, about whether qualitative findings that have been generated based on differing epistemological frameworks (e.g. grounded theory vs phenomenology) should be integrated, interpreted, and synthesized within systematic reviews (Finfgeld, 2003; Zimmer, 2006). First, reviewers are urged to remember that combining findings from different epistemological perspectives represents a form of triangulation and enhances validity (Finfgeld-Connett, 2010b). Second, qualitative findings, regardless of their epistemological origins, constitute evidence-based results that have the potential to add to qualitative systematic reviews. Ignoring selected findings solely due to their epistemological origins seems entirely contrary to the goal of knowledge development and theory generation, especially when the meaning and context of the original findings are preserved.
Data analysis and sample size
Some mention should be made of the practicality of the proposed content analysis methods in relationship to sample size. Undoubtedly, data analysis becomes more complicated as sample size increases. Of particular concern is the risk of becoming overwhelmed by redundancy and missing important nuances such as those related to fit. For this reason, discretionary sampling criteria that enhance topical inclusivity are encouraged, while excessive satiation of the findings is discouraged (Finfgeld-Connett, 2010b; Finfgeld-Connett and Johnson, 2013).
From 2005 to 2008, it is estimated that the median number of research reports per health care–related systematic review was 14, and the range was 2–113 (Hannes and Macaitis, 2012). Based on the author’s experience, the median number of reports (i.e. 14) can easily be managed using the content analysis methods that are described in this article (e.g. Finfgeld-Connett, 2009). That said, data management challenges would be anticipated with samples as large as 113. When managing samples of this size, reviewers may want to use qualitative data analysis software.
When deciding whether to use computer software, reviewers are urged to consider some potential disadvantages. In addition to the inherent threat that excessively large samples pose (Finfgeld-Connett, 2010b; Finfgeld-Connett and Johnson, 2013), the use of computer software can result in the loss of flexibility and oversimplification of the data analysis process. Scholars suggest that the outcome of this is likely to be mechanistic aggregation rather than full integration, interpretation, and synthesis of findings (James, 2012; Lempert, 2007; Major and Savin-Baden, 2010; Paterson et al., 2001; Stern, 2007). As such, reviewers who aim to build knowledge and generate theory are reminded that data analysis software is not a substitute for well-defined research topics and questions and expertly delimited samples.
Conclusion
Since the late 1980s, the number of qualitative systematic reviews has grown significantly, and they have assumed greater importance in terms of knowledge development and theory generation. These advances are threatened, however, by the lack of clarity that currently surrounds the data analysis methods that are used to conduct these investigations. This ‘black box’ needs to be eliminated to ensure further growth and development of this research genre (Hannes and Macaitis, 2012: 33). In the future, qualitative systematic reviewers are encouraged to use content analysis and to fully communicate how it was executed.
Although content analysis has been used in the past to conduct qualitative systematic reviews, until now, little has been written about how it should be adapted, especially for purposes of knowledge development and theory generation. Key differences between using content analysis for conducting primary qualitative research studies versus knowledge-development and theory-generating qualitative systematic reviews stem from the fact that the raw data that are used to conduct the latter is, by nature, highly systematized and contextualized. Due to this difference, qualitative systematic reviewers should avoid overmanipulating and processing the data (e.g. deconstructing and abstracting) for purposes of analysis.
When carrying out knowledge-building and theory-generating qualitative systematic reviews, reflective memoing and diagramming are highly important. As a result of these processes, findings are integrated, interpreted, and synthesized across studies, and interconnections between and among resultant elements are explicated. It is through reflective memoing and diagramming, in particular, that existing conceptualizations and theoretical frameworks are enhanced and knowledge development occurs.
Footnotes
Declaration of conflicting interests
The content is solely the responsibility of the author and does not necessarily represent the official views of the National Institutes of Health.
Funding
This work was supported by Grant Number R21DA024749 from the National Institute on Drug Abuse.