PROTOCOL: Critical appraisal of methodological quality and reporting items of systematic reviews with meta-analysis in evidence-based social science in China: A systematic review

BACKGROUND

Evidence-based practice has gradually developed, and more systematic reviews and meta-analyses have been conducted in the field of social sciences in China. Systematic reviews aim to summarize “the best available research on a specific question by synthesizing the results of several studies.” They use transparent procedures to search, evaluate, and synthesize the results of relevant research whilst minimizing bias. Because of this integrative effect, systematic reviews can provide more valuable references for policy-making and practice than individual studies. Therefore, for example, the Oxford Centre of Evidence-Based Medicine recommended systematic review/meta-analysis as Level A evidence (Level 1 evidence) in the evidence classification (Gates & March, 2016). The Growing What Works movement in the United Kingdom bases its evidence-based decision-making products on systematic reviews (Gough et al., 2018).

High-quality systematic reviews and meta-analyses can provide effective and scientific evidence for decision-makers, and they are also the primary source of information for researchers to quickly grasp the current progress of a research problem (Oliver & Dickson, 2015). However, the reporting quality of a systematic review depends on its methodological rigor and the clarity of the research report. Besides, differences in methods may lead to completely opposite conclusions of systematic reviews on the same research topic (Schalken & Rietbergen, 2017; Vrieze, 2018) and even mislead subsequent researchers.

In 1987, Mulrow evaluated 50 reviews published from June 1985 to June 1986 in four major medical journals (Annals of Internal Medicine, Archives of Internal Medicine, The Journal of the American Medical Association, and The New England Journal of Medicine) and found that none of them met all eight clear scientific criteria, such as evaluating the quality of the included studies (Mulrow, 1987), Meanwhile, Sacks et al. (1987) evaluated the reporting adequacy of 83 English-language meta-analyses of randomized controlled trials in the medical field that published from January 1966 through October 1986, involving in six areas: study design, combinability, control of bias, statistical analysis, sensitivity analysis, and application of results. The results showed that the quality of the reports from these reviews was low, with only 1–14 items of 83 meta-analyses being fully reported (Sacks et al., 1987). After 10 years, Sacks et al. (1996) updated the study finding that the situation had hardly improved (Sacks et al., 1996).

To improve the quality of meta-analysis, Moher et al. (1999) issued a guideline, named Quality of Reporting of Meta-Analysis (QUOROM), focusing on the report quality of meta-analysis on Randomized controlled trials (Moher et al., 1999). In 2009, they revised QUOROM guidelines and renamed them Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA), which also considered the quality of reporting on systematic reviews. With the advances in systematic review methodology and terminology, Page et al. (2021) developed the PRISMA 2020 statement for reporting systematic reviews based on 60 documents with reporting guidelines for systematic reviews to generate suggested modifications to the PRISMA 2009 statement in 2021 (Page et al., 2021). Meanwhile, the number of published meta-analyses concerning observational studies in health has increased substantially, Stroup et al. (2000) held a workshop in Atlanta Ga, in April 1997, and proposed a reporting checklist for Meta-analysis of Observational Studies in Epidemiology (MOOSE) to examine the reporting of meta-analyses of observational studies (Stroup et al., 2000).

In addition to the reporting checklist, Shea et al. (2009) developed a measurement tool to assess systematic reviews (AMSTAR) to evaluate the methodological quality of systematic reviews on randomized controlled trials (Shea et al., 2009). After receiving comments and feedback, the AMSTAR group revised AMSTAR and released AMSTAR-2 in September 2017 (Shea et al., 2017), which also included non-randomized studies of interventions (NRSI). In 2015, Diekemper et al. (2015) developed a Documentation and Appraisal Review Tool (DART) for systematic reviews, which explicitly included a quality review for biases specific to observational studies.

After these guidelines were released, more studies were conducted to explore the methodology and reporting quality of systematic reviews and meta-analyses in medical research, such as substance abuse (Kim et al., 2021), pediatrics (Bo et al., 2020), nursing (Jin et al., 2014), orthopedics (Gagnier & Kellam, 2013), and etc.

In recent years, systematic reviews and meta-analyses have been increasing in the field of social sciences. Social science is taken to mean any branch of academic study or science that deals with human behavior in its social and cultural aspects. It is mainly focused on the scientific study of human society and social relationships. Some researchers have already assessed the quality of systematic reviews in social science. For example, Kogut et al. (2019) examined the reporting quality of mathematics education systematic review with 40 reviews, they found deficiencies in search processes and reporting of search methods (Kogut et al., 2019). Wang et al. (2021) examined the reporting quality of 96 Campbell Systematic Reviews, finding that fewer than half (42%) were of high quality, but that quality had risen since standards were introduced (Wang et al., 2021).

In China, reviews are being undertaken in areas including Marxist Theory Studies, Management Science, Philosophy, Religion Studies, Linguistics, Law, Education, Economics, Geography, Ethnography, and Cultural Studies, Archaeology, History, Psychology, Sociology, Journalism, and Communication Studies, Political Science, Library Information Science, Sport, and Art.

Meanwhile, Chinese researchers introduced the critical appraisal methods with Chinese versions of critical appraisal tools (Ge et al., 2017; Li et al., 2009; Tao et al., 2018; Tian et al., 2015; Xiong & Chen, 2011; Zhan, 2010; Zhang et al., 2015). These tools have been widely used in systematic reviews in medical fields (Wang et al., 2015), and analyzed the methodology and reporting quality of Chinese reviews.

Tian et al. (2017) compared the methodological and reporting quality of 100 systematic reviews by authors from China and those from the United States and found them to be of similar quality (Tian et al., 2017). In 2022, Bai et al. randomly selected 200 Chinese systematic reviews in the social science field published from 2000 to 2019 in the Chinese Social Sciences Citation Index (CSSCI) database. They examined the methodological and reporting quality of these reviews and suggested that the quality of the systematic reviews was below the average level (Bai et al., 2022). However, the data source they searched was the CSSCI database which covers 500 of 2700 Chinese academic journals of social sciences, thus, it is reasonable to doubt possible selection bias in their review. Therefore, the present study aims to evaluate whether and to what extent reporting and methodology standards are met in the systematic reviews of social science in China and to assess the applicability of these tools in the Chinese context with content analysis.

OBJECTIVES

The present review includes three objectives: 1.

To evaluate the reporting quality of systematic reviews published in Chinese social science journals against the PRISMA and MOOSE standards.

METHODS