How to Write a Systematic Review

Meta-Analysis and Systematic Review

Although commonly used interchangeably, systematic reviews and meta-analyses are not the same. The Cochrane Collaboration defines a systematic review as a comprehensive high-level summary of primary research on a specific research question that attempts to identify, select, synthesize, and appraise all high-quality evidence relevant to that question to answer it. Further, systematic reviews collate all evidence pertinent to a priori selected criteria for eligibility to address the specific research question. Additionally, they identify and minimize bias via transparent, explicit, and systematic methodology.

A meta-analysis utilizes statistical methods (as differentiated from systematic reviews) to quantitatively evaluate pooled data from single studies. Individual studies are assigned a weight based on the sample size. Conclusions are reported based on the accuracy and precision (mean and confidence interval [CI] relative to a “zero effect” line on a forest plot) of individual studies’ results. The width of individual studies’ 95% CIs graphically depicts the degree of clinical, statistical, and methodological heterogeneity that is inevitably found in any clinical study. The forest plot conveys the “take-home point” conclusion of a meta-analysis in one simple figure. Familiarity in reading and interpreting a forest plot is essential in understanding the effect of the analyzed treatment(s) and their magnitudes. A meta-analysis does not necessarily mandate comprehensive inclusion of all studies relevant to a specific topic (eg, as in a systematic review), only the mathematical assimilation of studies. Thus, not all meta-analyses are systematic reviews. Likewise, not all systematic reviews are meta-analyses unless all studies are identified, included, and analyzed (systematic review) quantitatively (meta-analysis).

The QUORUM (Quality of Reporting of Meta-analyses) statement was established in 1996 by a group of epidemiologists, biostatisticians, medical editors, and researchers to improve the quality of conduct and reporting of systematic reviews and meta-analyses. ¹⁹ Developments in the performance and reporting of reviews led to the revision of QUORUM to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines and checklist in 2009. ¹⁷ The 27-item checklist was intended to improve review quality, not generate a review quality score. Incorporation of PRISMA guidelines in sports medicine and orthopaedic surgery literature is being increasingly recognized through support and promotion by high-quality publications. ³⁶ Further, PRISMA has been included as an integral component in the international EQUATOR (Enhancing the Quality and Transparency of Health Care Research) initiative to improve the reporting of published research. ¹¹

Getting Started

The PRISMA guidelines recommend open registration of all systematic reviews. ²⁰ Given the recent proliferation of systematic reviews and meta-analyses (Figure 2), this reduces the potential for duplication of resources devoted to a specific clinical topic. The Journal of Bone and Joint Surgery, Clinical Orthopaedics and Related Research, and Journal of Pediatric Orthopaedics have recently proposed an expectation of all authors of systematic reviews or meta-analyses to identify any similar reviews and justify why a new investigation is unique and illustrates different findings than prior reviews. ³⁶ More importantly, registration improves the quality of conduct of the review and its eventual reporting. Thus, on February 22, 2011, PROSPERO, the online, free, prospective international systematic review register, was launched for any and all health care–related research.^2,33 Nonpublication of finished systematic reviews, just as with clinical trials (publication bias), is a problem that may lead to overestimates of the effect of an intervention reported in the medical literature. ²⁹ Review registration is expected to reduce that bias, which preferentially selects the publication of only studies that have “positive” findings by ensuring public awareness before study commencement. Further, it has been demonstrated that studies with statistically significant positive findings are more likely to be reported in the English-language literature. ⁷ Thus, the journal publication’s language for included studies is a necessary source of bias that must be accounted for either before or during the review.

OPEN IN VIEWER

Figure 2.

According to the PubMed database, over the past 20 years, the number of systematic reviews and meta-analyses has increased each year.

Purpose of the Review: Identifying an Answerable Question(s)

The purpose of a systematic review should be to answer an important answerable clinical question or identify areas of high clinical importance that are underreported in the medical literature.^8,21 The question must be specific; however, it must not be too specific. If the posed question is too broad (eg, “Is exercise beneficial?”), then the reviewer is unable to properly focus the literature to a manageable number of studies to review and analyze. If the posed question is too narrow (eg, “Is exercising 43 minutes per day 3 times per week better than exercising 38 minutes per day 4 times per week?”), then there are not enough published reports available to answer the question. An appropriate question should be original and focused on the precise identification of the participant(s), intervention(s), comparison(s), outcome(s), and study design (PICOS criteria). A meticulous approach to the PICOS criteria is what makes a review “systematic,” contrary to a simple narrative review. It is the information within PICOS that establishes the inclusion and exclusion criteria for the studies analyzed.

Eligible Studies: Inclusion and Exclusion Criteria

Once the study purpose is identified and an answerable question posed, the reviewer must determine the study inclusion and exclusion criteria. Generation of a PRISMA flowchart (Figure 3) that demonstrates the identification and screening of potentially eligible studies determines the final number of studies included for analysis. Greater specificity of the inclusion criteria (improved review internal validity) limits the heterogeneity (review external validity) of the studies in the final analysis. However, depending on the study topic, greater study homogeneity is achieved at the expense of the number of studies (and patients) analyzed. A priori, these criteria should be established. However, it is not uncommon to need to alter criteria as the study search strategy ensues. For example, consider the following scenario: A novice reviewer asks the question “What is the rate of dislocation following total hip replacement?” The original inclusion criteria presented all clinical outcome studies reporting a minimum 2-year follow-up after total hip replacement. The search commences, and the reviewer encounters studies on different surgical approaches (eg, posterior, lateral, anterior). The researcher must now make a decision: (1) include all surgical approaches and combine their rates, (2) include all surgical approaches and report and compare the rates for each separate approach, or (3) only include one approach and report its individual rate. Preliminary research for the study background should develop enough of a knowledge base to set up an appropriate and thorough list of inclusion and exclusion criteria so that significant alterations are unnecessary during the course of the search, as shown in the total hip example.

OPEN IN VIEWER

Figure 3.

PRISMA flowchart that shows the step-by-step process of the application of inclusion and exclusion criteria to generate a final number of studies for analysis in the systematic review.

Critical evaluation of eligible studies also involves the assessment of the study’s level of evidence. ³⁵ A minimum level of evidence may be an inclusion criterion in a systematic review. The highest level of evidence should always be sought. However, important studies that evaluate the clinical question should not necessarily be excluded to meet the “highest” level of evidence. Sometimes, the better performed study with more accurate information may be provided in a “lower” level investigation. Leaving this information out to meet only “level I” studies may actually reduce the effectiveness of a well-performed systematic review. Further, depending on the relevant topic, it may be discovered that prospective randomized trials do not exist. In certain situations, level III and IV evidence may be the best possible evidence. For example, consider the following scenario: A reviewer asks the question “What are the clinical outcome differences between compartment syndrome of the leg treated with 4-compartment fasciotomy within 4 hours and within 24 hours?” Clearly, no researcher would ever randomize a patient to the latter group if the patient presented within 4 hours of the time of diagnosis. Only retrospective case series would be available to study this patient. Therefore, a systematic review on this topic is the same level of evidence as the minimum level of evidence of the studies that it analyzes. Nevertheless, the “best available evidence” does not always need to be a randomized controlled trial.

Primary and Exploratory Outcomes

Just as part of the inclusion and exclusion criteria are established a priori, the primary study outcomes should be ascertained before the studies are selected and analyzed. Primary end points of a systematic review should be analogous to those of a clinical trial. It is the measure upon which success or failure of an intervention is based (ie, acceptance or rejection of the null hypothesis). It is the main effect tested statistically for a difference and whether that difference is clinically meaningful. Beyond primary and secondary outcome measure(s) evaluated, exploratory end points are generally assessed post hoc. As with clinical trials, in a review, a priori planned outcome comparisons of an intervention on 2 groups of participants have a greater degree of statistical power compared with post hoc comparisons.

Search Methodology

The search strategy should be strictly focused on the PICOS criteria. Before beginning the search, the authors need to be aware that a PRISMA flowchart (Figure 3 and Appendices 1-3 [available in the online version of this article at http://ajsm.sagepub.com/supplemental]) should be created to illustrate study identification, screening, eligibility, inclusion, and analysis. There are several nonmutually exclusive electronic databases publicly available (free of charge) for the extraction of studies to be included in systematic review analyses. Pay-per-use databases are also accessible. Use of only a single database is insufficient.

PubMed is a free database that utilizes the MEDLINE database maintained by the United States National Library of Medicine of the National Institutes of Health. As of December 31, 2012, there are 22.4 million citations in the PubMed database. Embase is a subscription-based database maintained by Elsevier BV, with more than 25.2 million citations (as of May 13, 2012). Embase contains all MEDLINE records and 5 million citations not included in MEDLINE (including more than 2000 exclusively Embase-indexed journals). The Embase database is growing by more than 1 million citations annually. PEDro (Physiotherapy Evidence Database) is a free database maintained by the Centre for Evidence-Based Physiotherapy. It contains more than 23,000 randomized controlled trials, systematic reviews, and clinical practice management guidelines.

The Cochrane Library is a collection of 6 unique databases maintained by John Wiley & Sons Ltd. Included in the Cochrane Library is the Cochrane Database of Systematic Reviews (CDSR), which contains 7626 systematic reviews as of December 1, 2012. The reviews in the latter are excellent examples upon which to guide a novice author of systematic reviews. The Cochrane Central Register of Controlled Trials (CENTRAL) contains more than 680,000 controlled trials, most of which are concurrently found in MEDLINE, Embase, Cochrane Review Groups’ specialized registers, and hand-search results register. SciVerse Scopus is a subscription-based database that contains 41 million records of peer-reviewed journals, books, conferences, and scientific web pages.

Other commonly used databases for the identification of studies relevant to the investigation for systematic reviews include CINAHL (Cumulative Index to Nursing and Allied Health Literature), SPORTDiscus, and Google Scholar. Several publications have documented the necessity of utilizing at least 2 databases and, in certain circumstances, hand-searching important selected journals.^16,27,34 Nevertheless, in reference to sports medicine/orthopaedic surgery meta-analyses, recall rates (defined as the proportion of primary studies analyzed within orthopaedic meta-analyses that are indexed in either MEDLINE or Embase) of 90% and 81% for MEDLINE and Embase were obtained individually for all primary studies included in the meta-analyses, respectively. ²⁵ Combining MEDLINE and Embase increased recall rates to 91%, and the addition of Cochrane databases increased recall rates to 97%. ²⁵

Regardless of the database(s) used, the initial search’s specificity must not compromise the sensitivity. In other words, it is better to manually review more journal title(s), abstract(s), and full-text article(s) in the database than to be too specific and omit potentially inclusive studies. The entire “take-home point” of the review, its results and conclusions, is based on the studies that it analyzes. Thus, it is imperative to ensure that all of the relevant studies are included. It may take a longer amount of time to perform manual searching; however, this is absolutely necessary. In addition to studies identified from the database(s) searched, all reference lists from these studies should be analyzed for the potential inclusion of studies omitted from the initial search. A minimum of 2 reviewers should perform the initial study identification, secondary study screening, and final determination of eligibility and study inclusion. Although the inclusion of conference abstracts reduces publication bias, the bias introduced by the lack of a formal rigorous peer-review process (and the often-present differences between conference abstracts and final published articles) often warrants the exclusion of abstracts from high-quality systematic reviews.

Once the database review of study titles, abstracts, and full-text articles has commenced, the investigator must be sure to eliminate duplicate patient populations found in different studies. This is applicable when authors publish more than one article on the same group of patients, usually with different lengths of follow-up or with analysis and reporting of a different primary or secondary outcome. The easiest way to identify duplicate patient populations is in the article’s Methods section, which reports the dates of patient enrollment. If 2 separate studies with the same authors and the same intervention have overlapping dates of patient enrollment, then only one study may be inclusive. In this situation, the reviewer should select the study with the higher level of evidence, greater number of patients, longer follow-up, or more thorough reporting of the primary outcome of interest.

Extraction and Analysis of Study Data: Reporting the Results

Once all exclusion criteria are applied and the final list of studies is identified for analysis, there are several effective methods of extraction of study data into a coherent group of PICOS parameters. These data collection forms can be either written paper checklists or electronic spreadsheets. Although a generic template form may be used to begin data extraction from included studies for a systematic review, each written review requires the individualization of data extracted to the topic of interest. Data can be input into a custom spreadsheet such as Microsoft Excel (Microsoft Corp, Redmond, Washington) (see Table 1 for an example of parameters sought and extracted). Multiple different checklists have been published by various academic institutions and private groups. Review authors should find one that suits their research style, research question, and outcomes of interest most appropriately.

OPEN IN VIEWER

Table 1

Relevant Information Sought to Be Extracted From Individual Trials ^a

Study details

Study author(s)

Journal

Year of publication

Years of patient enrollment

Presence, absence, or lack of reporting of financial COI

Level of evidence

Study design (RCT, prospective, retrospective, etc)

Single vs multicenter

Country of study performance

Key study statements

Primary and secondary purposes

Hypothesis(es)

Study inclusion and exclusion criteria

Intended primary and secondary outcome measures

Evaluation of study methodological quality

Cochrane Bone, Joint and Muscle Trauma Group’s methodological quality score

CLEAR-NPT

Delphi list

Detsky Quality Assessment Scale

Coleman (and modified) methodology score

Quality appraisal tool

Jadad score

CONSORT

STROBE statement

Newcastle-Ottawa Quality Assessment Scale

Participants

Number of patients enrolled

Blinded? Yes/no

Number of male/female patients

Number of body parts analyzed (eg, knees)

Number of right vs left sides

Number of dominant vs nondominant

Number of surgical interventions (eg, total knee arthroplasty)

Patient age, mean ± SD

Clinical follow-up, mean ± SD

% follow-up (eg, >80%)

Radiographic follow-up (if applicable), mean ± SD

Duration of symptoms warranting an intervention, mean ± SD

Length of treatment before intervention, mean ± SD

Number of prior surgical procedures, mean ± SD

Other treatments performed

Interventions

Number of interventions performed in each group, mean ± SD

Number of providers performing the intervention

Blinded? Yes/no

Comparators

Number of patients in control or comparator groups

Description of “treatment” in the control or comparator groups

Outcomes

Explicit description of all outcome measures and scores utilized

Preintervention score

1-year score

2-year score

5-year score

Etc

Number of complications

Number of reinterventions

Independent blinded evaluator/assessor?

Were study conclusions based on study results?

a

CLEAR-NPT, Checklist to Evaluate and Report a Non-Pharmacologic Trial; COI, conflict of interest; CONSORT, Consolidated Standards of Reporting Trials; RCT, randomized controlled trial; SD, standard deviation; STROBE, Strengthening the Reporting of Observational Studies in Epidemiology.

Two commonly used checklists to assist with the extraction of trial details are publicly available from the Centre for Evidence-Based Medicine (CEBM) ³¹ and the Cochrane Collaboration. ⁴ In 2005, Spindler et al ²⁶ were the first to publish a checklist for use in the conduct and reporting of systematic reviews in orthopaedic surgery (4 written pages). It is not uncommon to continue the application of review exclusion criteria during data extraction. Study details not revealed in the title or abstract are revealed upon review of the full text and may warrant exclusion.

Statistical Assimilation of Data (Meta-Analysis)

Once all studies’ data have been extracted, a brief tabular narrative of each investigation may be presented for publication within the article. ¹² Columnar data usually include the (1) year of publication, (2) lead author, (3) number of study participants, (4) participant group(s), (5) intervention(s), (6) follow-up period, and (7) outcomes. Unique study details may warrant the addition of further study details in a tabular/graphical form. ¹⁴ Additional tables may be added to illustrate complications and reoperations ¹⁴ and study limitations/biases. ¹³ After each study has been critically evaluated, the decision must be made to quantitatively group the data with like outcome tools together (meta-analysis). ¹⁵ Homogeneous studies allow for the assimilation of separate studies’ results and the creation of a forest plot, which is an illustration that displays the relative strength (effect size) of an individual study’s results that evaluate the same intervention with the same outcome measure. Effect size analysis allows for a direct numerical comparison between different studies. Performance of a meta-analysis is not always possible because of heterogeneity among studies. Heterogeneity may be assessed in 1 of 2 ways. One is visually with the “eyeball test,” published by the CEBM in 2005. ³¹ This test simply seeks to find the overlap of CIs of the trials with the summary estimate on the forest plot. A second method involves a quantitative assessment of statistical heterogeneity via tests such as the Cochrane Q (χ²) test, index test (I²), or τ² test (or T²). It is recommended to seek the assistance of a biostatistician who is experienced in meta-analyses early in the timeline of the investigation. ³⁷ Additionally, it is important to understand, identify, and directly report the difference between statistical significance and clinical relevance within the individual study’s findings. For example, the postoperative outcome difference between 2 new surgical techniques for cartilage repair is statistically significant with regard to the International Knee Documentation Committee (IKDC) subjective score. However, the effect size of this difference is small, not meeting the threshold of minimally detectable change or minimal clinically important difference. ¹⁰ Therefore, although the difference is statistically significant, the difference is not clinically large enough to be perceived by the patient as different. ¹⁸ These differences are key in the ability to report whether patients have improved (treatment response/responders), whether that improvement is “back to normal,” ¹⁸ and whether the outcome is acceptable to the patient (patient acceptable symptom state [PASS]). ³⁰

Evaluation of Study Methodological Quality

The quality of a systematic review (Table 2) is only as good as the studies that it analyzes. Thus, a review of only randomized controlled trials with level I evidence is a level I review. Further, a review of multiple level I randomized trials and multiple level III retrospective case comparison studies is a level III review. In addition to level of evidence ratings (updated by the CEBM in 2011), ³² there are several different study methodological quality scores available to numerically grade the quality of a trial (Table 1). Some questionnaires are designed as guides to help improve the conduct and reporting of trials and are not intended to generate a quantitative result (eg, CONSORT). When intended to numerically grade the quality of an investigation, these scores simply grade the quality of study reporting and not necessarily the quality of study performance and conduct. It is important to recognize that, just as the review search and study selection be performed by at least 2 reviewers, it is also important to assess study quality with at least 2 reviewers as well. These quality evaluation tools describe the potential sources of bias within studies (eg, selection, performance, transfer, detection, publication, study design). The reviewers must be aware that these steps are critical to grading the strength of evidence. Therefore, they must be accurate, and reviewers may note that this step in the review takes a significant length of time, depending on how many studies are being analyzed. Other similar assessment tools for grading the evidence include SORT (Strength of Recommendation Taxonomy) ⁶ and GRADE (Grading of Recommendations Assessment, Development, and Evaluation). ⁹

OPEN IN VIEWER

Table 2

Pearls and Pitfalls for the Conduct of a High-Quality Systematic Review ^a

Pearls	Pitfalls
Answerable question	Underestimating the length of time to complete the review
Does the review improve significantly upon existing reviews?	Failing to identify if the review has already been conducted recently
PICOS strategy protocol	Question nonspecific or too broad (“unanswerable”)
PROSPERO registration	Failure to identify explicit study inclusion and exclusion criteria
PRISMA guidelines, checklist, and flowchart	Failure of review “transparency”
Thorough data extraction	Not excluding duplicate study populations within different studies
Quantitative synthesis of studies’ data (if applicable, meta-analysis)	Failure to recognize and report heterogeneity between studies
Grade the evidence and strength of recommendations (eg, SORT, GRADE)	Failure to recognize and report studies’ bias
Explicit statement of the review’s “take-home point”	Making claims in conclusions that are beyond the facts/results of the review

a

GRADE, Grading of Recommendations Assessment, Development, and Evaluation; PICOS, Participants, Interventions, Comparisons/Controls, Outcomes, and Study Design; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-analyses; SORT, Strength of Recommendation Taxonomy.

Beyond the analysis of individual study quality, questionnaires exist to grade the quality of systematic reviews and meta-analyses as well. Therefore, these assessment tools may be used by authors of reviews to guide them in the relevant steps of performance and reporting of the review. Evaluation of systematic review methodological quality via AMSTAR (assessment of multiple systematic reviews) has recently been introduced to improve the conduct and reporting of systematic reviews. ²³ The Cochrane Collaboration has recently introduced the MECIR (Methodological Expectations of Cochrane Intervention Reviews) guidelines, which list 80 ³ and 108 ²⁸ items for conducting ³ and reporting ²⁸ systematic reviews, respectively. The MECIR guidelines complement and supplement PRISMA guidelines. They explicitly ask whether an item relevant to an investigation is either mandatory or highly desirable.

Summarize the Findings

The final, and most important, step in the systematic review is the reporting of the “take-home point.” Often, busy clinicians only have time to read an abstract or the study conclusions. This limited amount of text is the authors’ single opportunity to convey the key findings of the review. Therefore, authors must address their answerable question(s) and whether their hypothesis or hypotheses were confirmed. Authors must also acknowledge the limitations identified in the analyzed studies, as this bias parallels the bias of the review. The authors must make conclusions based on the review’s results. There is no room for speculation or discussion in this section, only the facts. The review’s conclusions should, in essence, be the answer to the following question: “If a reader were to remember one thing about my review, what would it be?”