Citation of prior research has increased in introduction and discussion sections with time: A survey of clinical trials in physiotherapy

Abstract

Background/aims:

Many clinical trials are reported without reference to the existing relevant high-quality research. This study aimed to investigate the extent to which authors of reports of clinical trials of physiotherapy interventions try to use high-quality clinical research to (1) help justify the need for the trial in the introduction and (2) help interpret the trial’s results in the discussion.

Methods:

Data were extracted from 221 clinical trials that were randomly selected from the Physiotherapy Evidence Database: 70 published in 2001 (10% sample) and 151 published in 2015 (10% sample). The Physiotherapy Evidence Database score (which rates methodological quality and completeness of reporting) for each trial was also downloaded.

Results:

Overall 41% of trial reports cited a systematic review or the results of a search for other evidence in the introduction section: 20% for 2001 and 50% for 2015 (relative risk = 2.3, 95% confidence interval = 1.5–3.8). For the discussion section, only 1 of 221 trials integrated the results of the trial into an existing meta-analysis, but citation of a relevant systematic review did increase from 17% in 2001 to 34% in 2015. There was no relationship between citation of existing research and the total Physiotherapy Evidence Database score.

Conclusion:

Published reports of clinical trials of physiotherapy interventions increasingly cite a systematic review or the results of a search for other evidence in the introduction, but integration with existing research in the discussion section is very rare. To encourage the use of existing research, stronger recommendations to refer to existing systematic reviews (where available) could be incorporated into reporting checklists and journal editorial guidelines.

Keywords

Clinical trial systematic review

Introduction

Science, including clinical research evaluating the effects of healthcare interventions, is cumulative.^1,2 Because of this, what is known and what is unknown should be evaluated before a clinical trial is conducted.^3–5 This information should also be summarised in the introduction section of the published report of the trial so that readers can understand why the trial needed to be done. For example, if there is no previous research to answer the trial’s question that helps to justify the need for the trial. Alternatively, if similar trials already exist, discussion of them can explain the reason why additional data are needed. This approach is a requirement of the Declaration of Helsinki, which states that medical research must ‘be based on a thorough knowledge of the scientific literature’.⁶ The results of a trial must also be interpreted with reference to existing clinical research in order to position the results in the totality of the available evidence.^7,8 For example, if the results of a new trial are different to the results of an existing trial, the possible reasons should be discussed to help the reader understand the new overall body of evidence. The importance of reporting the results of previous research in the introduction and discussion sections is evidenced by 2 of the 25 items in the Consolidated Standards of Reporting Trials (CONSORT) checklist relating to this issue: item 2a is ‘Scientific background and explanation of rationale’ in the introduction section and item 22 is ‘Interpretation consistent with results, balancing benefits and harms, and considering other relevant evidence’ in the discussion section.^9,10

A series of articles in The Lancet has highlighted the need to increase value and reduce waste in biomedical research.^11–15 One way that waste can occur in research is that trials of interventions are performed when the effect of the intervention is already clear from previous research, and the pooled estimate of the effect is unlikely to change with further trials.¹⁶ Combining the results of new research with the results of existing research, for example, in meta-analyses, increases the value of new research.¹⁷ Therefore, researchers should ensure that the design of a new clinical trial is informed by relevant existing evidence, and the results of a new clinical trial are incorporated into the existing evidence.¹⁴ Readers should also hope to see this information in the article. Therefore, high-quality clinical research should be cited, where possible, in the introduction section of a trial report to indicate that existing research informed the design of the study. Citation in the discussion section would indicate that trial results were being integrated into existing evidence.

When seeking to summarise existing evidence on the same topic as a trial, an efficient strategy is to cite a systematic review, which is a literature review that uses explicit methods to comprehensively identify and pool all the existing evidence while minimising the potential for bias in its results.¹⁷ Therefore, where available, a systematic review would be an appropriate article to cite in the introduction and discussion sections to help summarise what is already known on the topic. In the discussion section, an additional possibility is to modify a forest plot from the cited systematic review by simply including the result of the new study. Note that this does not update the systematic review, which would require new searches, data extraction and meta-analysis – essentially, a whole new article. What it does indicate, however, is how the estimate from the new trial compares to previous trials identified in a systematic review and how the new trial changes that review’s overall estimate.

An analysis of five highly cited medical journals (Annals of Internal Medicine, BMJ, The Lancet, JAMA and New England Journal of Medicine) revealed that systematic reviews were rarely cited in the introduction and discussion sections of reports of clinical trials.^2,18,19 Surprisingly, there was not consistent improvement in the citation of systematic reviews in the discussion section over time. The percentage of trial reports citing systematic reviews in the discussion section was 23% for trials published in May 1997,¹⁸ 9% for trials published in May 2001² and 28% for trials published in May 2005.¹⁹ Only 8% of trials published in 1997, but none in 2001 or 2005, integrated their results with existing research.^2,18,19 About one-quarter (27%) of trial reports published in May 2005 cited a systematic review in the introduction section.¹⁹ An examination of a larger set of 1523 articles reporting the results of trials demonstrated a failure to cite previous clinical trials, with the prior research citation index (i.e. the number of cited clinical trials divided by the number of clinical trials available to cite) being 0.21 (95% confidence interval = 0.18–0.24).⁷ That is, articles reporting trial results cited less than one-quarter of the existing clinical trials addressing the same research question. No trials integrated their results into existing research in the discussion section. While it is likely that articles reporting the results of clinical trials evaluating physiotherapy interventions also inadequately cite existing research, this has not been specifically investigated.

None of the surveys of citation in articles reporting the results of clinical trials examined the relationship between the methodological quality of a trial report and whether it cites relevant existing research. However, a relationship might be anticipated; researchers who address the recommendations of the Declaration of Helsinki and the CONSORT statement thoroughly may be more likely to thoroughly address issues such as the control of bias and the completeness of reporting. One aspect of this issue has recently been examined: the relationship between reporting of interventions (evaluated using the Template for Intervention Description and Replication (TIDieR) checklist)²⁰ and methodological quality (evaluated using the Physiotherapy Evidence Database scale)²¹ for clinical trials in physiotherapy.²² There appeared to be no clear relationship between the completeness of reporting of interventions and design features. Nevertheless, the relationship between methodological quality and the citation of relevant existing clinical research should be examined.

The aim of this study was to investigate the extent to which published reports of randomised controlled trials evaluating the effects of physiotherapy interventions use high-quality clinical research to justify the need for the trial in the introduction section and interpret the trial’s results in the discussion section. A secondary aim was to determine whether citation of high-quality clinical research in the introduction and discussion sections has increased over time by comparing trial reports published in 2001 with those published in 2015. The relationship between methodological quality and citation of existing research will also be evaluated by comparing the total Physiotherapy Evidence Database score of trial reports that do and don’t cite existing research in their introduction and discussion sections.

Material and methods

Study selection

A representative sample of 10% (or approximately 200) of clinical trials indexed on the Physiotherapy Evidence Database (www.pedro.org.au) was evaluated for the publication years 2001 and 2015. These years were selected because trials published in 2001 were evaluated in a survey of the medical literature,² and 2015 was the most recent full year. This sample size provides a 95% confidence interval of ±0.05, given an estimated prevalence of 0.15 for the primary outcome (based on the prevalence observed in similar studies in medical journals).^2,18,19

The Physiotherapy Evidence Database was selected as the source of trial reports because it is one of the most complete indexes of reports of physiotherapy trials,^23,24 and all trial reports are assessed for methodological quality using the Physiotherapy Evidence Database scale.²¹ This scale is an 11-item instrument used to rate criteria related to internal validity and completeness of reporting (eligibility criteria and source; random allocation; concealed allocation; similarity at baseline; subject, therapist and assessor blinding; >85% follow-up; intention-to-treat analysis; between-group statistical comparison; and provides point and variability measures). Of the items, nine are based on the Delphi list.²⁵ Because one of the items from the Delphi list relates to external validity (i.e. eligibility criteria and source), it is excluded from the total score, which is the number of items met (range 0 (low quality) to 10 (high quality)). Rating is carried out by two independent raters, with disagreements resolved by discussion or, if necessary, arbitrated by a third rater.²⁶ The Physiotherapy Evidence Database scale is reliable,²¹ valid,²⁷ and the items can be summed to produce the total score that can be treated as an interval level measurement.²⁸

Trial reports were included if they were the primary report of the trial results (i.e. secondary analyses were excluded), if the year of publication was in 2001 or 2015 and if the trial had complete Physiotherapy Evidence Database coding and rating (i.e. were not in process). There were no restrictions for the area of physiotherapy practice or the language in which the trials were published.

The citation details (authors, title, journal, year, volume, issue, pages), area of physiotherapy (Physiotherapy Evidence Database sub-discipline codes: cardiothoracics, continence and women’s health, ergonomics and occupational health, gerontology, musculoskeletal, neurology, oncology, orthopaedics, paediatrics and sports) and Physiotherapy Evidence Database scale (total score) were downloaded from the Physiotherapy Evidence Database for all trials published in 2001 and 2015. The random number function in Excel was used to select the random sample for each year. Full-text copies of each trial report were retrieved for these samples.

Data extraction

Two investigators independently confirmed eligibility and, for eligible trials, extracted the data using a customised Excel spreadsheet. Any disagreements were resolved by discussion. When reports were written in languages other than English, confirmation of eligibility and data extraction were performed by bilingual colleagues.

Data extracted from the ‘Introduction’ section of each randomly selected trial report were as follows:

Was a systematic review relevant to the question being evaluated in the trial report cited (coded as ‘yes’ or ‘no’);

If the answer to question 1 was ‘no’, does the trial report state that evidence was searched for and summarise the results of that search (coded as ‘yes’, ‘no’ or ‘not applicable’ (when the answer to question 1 was ‘yes’));

If the answers to questions 1 and 2 were ‘no’, does the trial report summarise the results of some evidence (coded as ‘yes’, ‘no’ or ‘not applicable’ (when an answer to questions 1 or 2 was ‘yes’));

If the answer to question 1 was ‘yes’, the year of publication of the most recent systematic review cited was recorded.

For question 1, only evidence-based clinical practice guidelines (i.e. based on systematic reviews of clinical trials or reports of clinical trials) or systematic reviews of clinical trials were applicable. For questions 2 and 3, evidence refers to randomised controlled trials. Guidelines, reviews and trials were considered to be relevant to the question evaluated in the included trial if the population matched and a similar intervention was used (however, the intervention could be broader than in the included trial). To fulfil question 2, words indicating that a search was performed (e.g. ‘search’) had to be included in the text. The primary outcome for this analysis of the introduction section was questions 1 and 2 combined (i.e. either a systematic review was cited or the results of a search for other evidence was stated).

A similar procedure was performed for the discussion section of the included trial reports. The data extracted were as follows:

Was a systematic review relevant to the question being evaluated in the trial report cited (coded as ‘yes’ or ‘no’);

If the answer to question 1 was ‘yes’, did the authors update the systematic review to integrate the new results from their trial (coded as ‘yes’, ‘no’ or ‘not applicable’ (when the answer to question 1 was ‘no’));

If the answer to question 1 was ‘no’, does the trial report indicate that evidence was searched for but not found (coded as ‘yes’, ‘no’ or ‘not applicable’ (when the answer to question 1 was ‘yes’));

If the answer to questions 1 and 3 were ‘no’, does the trial report summarise the results of some evidence (coded as ‘yes’, ‘no’ or ‘not applicable’ (when the answer to questions 1 or 3 were ‘no’));

If the answer to question 4 was ‘yes’, did the authors integrate the new results from their trial with this other evidence (coded as ‘yes’, ‘no’ or ‘not applicable’ (when the answer to questions 1 or 3 were ‘yes’ or the answer to question 4 was ‘no’)).

The results from the trial were considered to be integrated with existing systematic reviews or trials if there were numerical data (e.g. mean difference and 95% confidence interval from meta-analysis). The primary outcome for this analysis of the discussion section was questions 2 and 5 combined (i.e. the authors integrated their results with the results of a systematic review or other evidence).

Data synthesis and analyses

To determine whether authors of clinical trials evaluating the effects of physiotherapy interventions attempt to use high-quality clinical research to justify the need for the trial in the introduction section and interpret the trial results in the discussion section (i.e. primary aims), the percentage and 95% confidence interval were calculated for the primary outcomes from the introduction (either a systematic review or the results of a search for other evidence was cited) and discussion (the authors integrated their results with a systematic review or other evidence) sections. Percentages were calculated for all datapoints extracted from the introduction and discussion sections. For trial reports that cite a relevant systematic review in the introduction section, the mean (standard deviation) number of years between publication of the cited systematic review and the trial report was calculated.

Two analyses were performed for the secondary aim of whether citation of systematic reviews in the introduction and discussion sections increased over time. Relative risk and 95% confidence interval were calculated for the proportion of trials citing systematic reviews or the results of a search for other evidence in the introduction section in 2001 versus 2015. The same analysis of 2001 versus 2015 was performed for the proportion of trials integrating the new results with a systematic review or other evidence in the discussion.

A two-sample t-test was used to compare the total Physiotherapy Evidence Database score of trial reports that did or didn’t cite relevant systematic reviews or the results of a search for other evidence in the introduction section. The same analysis was performed for the scores of trial reports that did or didn’t integrate the results with a systematic review or other evidence in the discussion section.

The Physiotherapy Evidence Database confidence interval calculator was used to calculate the 95% confidence intervals for the primary analysis,²⁹ and Statistical Package for the Social Science (version 23, 2015) was used for all other statistical analyses. p values < 0.05 were considered statistically significant.

Results

In the 4 April 2016 update of the Physiotherapy Evidence Database, there were 33,045 records, of which 26,358 were clinical trials. There were 708 trial reports published in 2001, from which 693 had complete rating and coding. In 2015, 1578 trial reports were published, and 1496 of these had complete rating and coding. A total of 70 trials were randomly selected from the 2001 sample and 151 from 2015. The flow of trials is illustrated in Figure 1.

Figure 1.

Flow of trials through the study.

The trials in the data set were from a range of sub-disciplines, mainly published in English and were of fair (4 or 5 of 10) to good (5–8 of 10) quality,³⁰ on average (see Table 1). The majority of trials were from the musculoskeletal sub-discipline (31%) followed by neurology and cardiothoracics (both 19%). The smallest number of trials were from oncology (3%) and ergonomics and occupational health (3%). A total of 92% of trials were published in English. Other languages were Chinese (10 trials), Portuguese (2), German (2), French (1), Spanish (1) and Korean (1). The mean total Physiotherapy Evidence Database score was 5.4 (standard deviation = 1.5) of 10 points.

Table 1.

Sub-disciplines of physiotherapy, languages the trials were published in and the total Physiotherapy Evidence Database score.

Variable	2001 (N = 70)	2015 (N = 151)	2001 + 2015 (N = 221)
Sub-discipline of physiotherapy
Cardiothoracics	17 (24%)	24 (16%)	41 (19%)
Continence and women’s health	9 (13%)	22 (15%)	31 (14%)
Ergonomics and occupational health	1 (1%)	5 (3%)	6 (3%)
Gerontology	9 (13%)	17 (11%)	26 (12%)
Musculoskeletal	24 (34%)	44 (29%)	68 (31%)
Neurology	7 (10%)	34 (23%)	41 (19%)
Oncology	1 (1%)	6 (4%)	7 (3%)
Orthopaedics	10 (14%)	4 (3%)	14 (6%)
Paediatrics	5 (7%)	15 (10%)	20 (9%)
Sports	6 (9%)	6 (4%)	12 (5%)
Other	3 (4%)	12 (8%)	15 (7%)
Language
English	61 (87%)	143 (95%)	204 (92%)
Other	9 (13%)	8 (5%)	17 (8%)
Total score (/10), mean (standard deviation)	4.9 (1.4)	5.6 (1.5)	5.4 (1.5)

Each trial can be coded under up to three sub-disciplines, so the percentages in the table for the sub-disciplines don’t add up to 100%. Data are number (%) unless otherwise specified.

The number of clinical trial reports using high-quality research to justify the need for the trial was relatively low. Overall 91 of 221 trial reports (41%, 95% confidence interval = 35%–48%) cited either a systematic review (41%) or the results of some evidence (38%) in the introduction (see Table 2, primary outcome for 2001 + 2015 combined). Only one trial report explicitly stated that a search for evidence was performed. Among the trial reports that cited a systematic review in the introduction, the mean difference between the publication date of the systematic review and the publication date of the trial report was 4 years (standard deviation = 3 years).

Table 2.

Data extracted from the introduction section of the included trial reports and the calculated primary outcome (shaded in grey) for the introduction section.

Variable	2001 (N = 70)	2015 (N = 151)	2001 + 2015 (N = 221)
1. Was a systematic review relevant to the trial cited?
Yes	14 (20%)	76 (50%)	90 (41%)
No	56 (80%)	75 (50%)	131 (59%)
2. Was evidence searched for and summarised?
Yes	1 (1%)	0 (0%)	1 (0.5%)
No	55 (79%)	75 (50%)	130 (59%)
Not applicable (when question 1 was ‘yes’)	14 (20%)	76 (50%)	90 (40.5%)
3. Were the results of some evidence summarised?
Yes	35 (50%)	49 (33%)	84 (38%)
No	20 (29%)	26 (17%)	46 (21%)
Not applicable (when question 1 or 2 was ‘yes’)	15 (21%)	76 (50%)	91 (41%)
4. Mean (standard deviation) years between publication of the cited systematic review and the trial report (if question 1 is ‘yes’)	5 (3)	4 (3)	4 (3)
Introduction primary outcome (questions 1 and 2 combined)
Yes	15 (21%)	76 (50%)	91 (41%)
No	55 (79%)	75 (50%)	130 (59%)

Data are number (%) unless otherwise specified.

There was virtually no integration of trial results with a systematic review or other evidence in the discussion section. While 64 (29%) of trials cited a systematic review and 108 trials (49%) cited some evidence, only 1 of 221 trial reports numerically combined the trial’s results into existing research (0.5%, 95% confidence interval = 0.1%–2.5%; see the discussion primary outcome in Table 3). No trial reports explicitly stated that a search for evidence was performed.

Table 3.

Data extracted from the discussion section of the included trial reports and the calculated primary outcome (shaded in grey) for the discussion section.

Variable	2001 (n = 70)	2015 (N = 151)	2001 + 2015 (N = 221)
1. Was a systematic review relevant to the trial cited?
Yes	12 (17%)	52 (34%)	64 (29%)
No	58 (83%)	99 (66%)	157 (71%)
2. Were the trial results integrated with the systematic review?
Yes	0 (0%)	1 (1%)	1 (0.5%)
No	12 (17%)	51 (34%)	63 (28.5%)
Not applicable (when question 1 was ‘no’)	58 (83%)	99 (66%)	157 (71%)
3. Was evidence searched for but not found?
Yes	0 (0%)	0 (0%)	0 (0%)
No	58 (83%)	99 (66%)	157 (71%)
Not applicable (when question 1 was ‘yes’)	12 (17%)	52 (34%)	64 (29%)
4. Were the results of some evidence summarised?
Yes	38 (54%)	70 (46%)	108 (49%)
No	20 (29%)	29 (19%)	49 (22%)
Not applicable (when question 1 or 3 was ‘yes’)	12 (17%)	52 (34%)	64 (29%)
5. Were the trial results integrated with other evidence?
Yes	0 (0%)	0 (0%)	0 (0%)
No	38 (54%)	70 (46%)	108 (49%)
Not applicable (when question 1 or 3 was ‘yes’)	32 (46%)	81 (54%)	113 (51%)
Discussion primary outcome (questions 2 and 5 combined)
Yes	0 (0%)	1 (1%)	1 (0.5%)
No	70 (100%)	150 (99%)	220 (99.5%)

Data are number (%).

Citation of high-quality clinical research in the introduction and discussion sections did increase with time. Citation of either a systematic review or the results of a search for other evidence in the introduction increased from 21% in trials published in 2001 to 50% for trials published in 2015 (see Table 2). This increase was statistically significant (relative risk 2.3, 95% confidence interval = 1.5–3.8). This influence of time was not apparent for the primary outcome in the discussion section (i.e. integration of trial results with a systematic review or other evidence), with 0% in the 2001 sample and 1% in 2015. Statistical analysis of the influence of time was not performed because of the low prevalence. However, citation of a relevant systematic review in the discussion section did increase (from 17% in 2001 to 34% in 2015; see Table 3).

There was no relationship between the quality of trials and the citation of high-quality evidence. The average total Physiotherapy Evidence Database score for the trials citing a systematic review or the results of a search for other evidence in the introduction was 5.5 of 10 (standard deviation = 1.5), compared to 5.3 (standard deviation = 1.4) for trials that did not do so. This difference was not statistically significant (t = −1.047, p = 0.296). The total score for the one trial where the results were integrated into the existing evidence in the discussion section was 8 of 10, compared to an average of 5.4 (standard deviation = 1.5) for trials that did not integrate their results. Because of the low prevalence of trials with the primary outcome for the discussion section, no statistical analysis was performed.

Discussion

While the citation of relevant high-quality clinical research in articles reporting the primary results of clinical trials of physiotherapy interventions has improved with time, there is still room for improvement. In their introduction section, 41% (95% confidence interval = 35%–48%) of the analysed trials either cite a systematic review or state that evidence was sought. Almost no trials (0.5%, 0.1%–2.5%) integrate their results with a systematic review or other evidence in the discussion section. There was a 2.3-fold increase in citation of existing evidence in the introduction between 2001 (21%) and 2015 (50%). While numeric integration with existing research in the discussion section did not change (0% in 2001 and 1% in 2015), citation of relevant systematic reviews increased from 17% in 2001 to 34% in 2015. There was no relationship between the methodological quality of the trial and the citation of evidence.

While we contend that adding the result of a new trial to the meta-analysis from an existing systematic review is an efficient and objective way to incorporate the new evidence into the existing evidence, we raise a number of caveats. Obviously, if the trial is novel (in the sense that no previous trials have compared the same interventions and measured a common outcome), then this approach will not be possible. The authors who choose to add their new trial’s result to the meta-analysis of an existing systematic review should be explicit in the discussion section that this is what they have done. In this way, it will be clear to readers that the revised forest plot shows them how the estimate from the new trial compares to previous trials identified in a systematic review and how the new trial changes that review’s overall estimate. Furthermore, it will be clear that the authors have not updated the entire review by repeating searches, data extraction and so on.

The main strengths of our study were the rigorous processes used for data extraction and the use of a large representative sample. All data were extracted by two independent raters, with any disagreements resolved by discussion. The trial reports examined in this study were randomly selected from a defined population (all physiotherapy trials published in 2001 and 2015 and indexed on the Physiotherapy Evidence Database). This is likely to mean that the results of our study are generalisable to all trials of physiotherapy intervention because the Physiotherapy Evidence Database has extremely high coverage of such trials.^23,24

Our sample of trials (N = 221) was substantially larger than previous studies examining citation of existing high-quality clinical research in medical trials (N = 18–33).^2,18,19 Using similar methods, we were able to establish that more physiotherapy trials cite relevant systematic reviews in their introduction (41%) and discussion (29%) sections compared with medical trials (27% and 18%, respectively).^2,18,19 In addition to the increase in citation of systematic reviews in the discussion section with time (from 9% in 2001 to 28% in 2005 for medical trials and from 17% in 2001 to 34% in 2015 for physiotherapy trials), we were able to quantify an increase in the citation rate of systematic reviews in the introduction section (from 20% in 2001 to 50% in 2015). One finding that is consistent between our study and the surveys of medical trials is that the prevalence of quantitative integration of trial results with an existing systematic review or other evidence in the discussion section is very low (0.5% for physiotherapy trials and 3% for medical trials).^2,18,19 In our sample, only one trial integrated the results from the trial into a meta-analysis from the previous Cochrane systematic review of acupuncture for acute stroke.³¹ A forest plot was used to present the integrated results.

Limitations of our study include the following: not evaluating the availability of relevant high-quality clinical research available for citation, limiting our evaluation to the primary reports of trial results and sampling from two discrete years. We did not determine whether there was a relevant systematic review or published trial available for citation for the included trials that did not cite existing research in their introduction (21%) or discussion (22%) sections. However, we did note whether the authors of each trial report commented on existing evidence, by citing an existing review, citing existing individual trials or stating that they searched for existing reviews/trials and could not find any. Any of these scenarios indicates that the authors of the trial report tried to put their trial in the context of existing evidence. This issue has also been investigated in a sample of 1523 published trials cited in 227 systematic reviews, with published trials citing less than one-quarter of the existing trials addressing the same research question.⁷ It is likely that our sample of trials would have similar results. The relatively low citation of systematic reviews in the introduction section (i.e. 50% in 2015) could be explained by justification for the need for the trial being provided in the planning stages for the trial, including in the grant application, ethics application, protocol and trial registration. A potential focus of future research could be to examine the citation of systematic reviews in other trial documents and compare citation rates between these documents and the primary reports of trial results. Finally, we could not evaluate the time course of the increase in citation in the introduction and discussion sections because we only sampled trials from two 1-year periods, separated by 15 years. Exploration of the time course of citation may help determine whether citation is related to the rapid increase in the number of systematic reviews which has occurred since 2001.³²

There is room for improvement in the citation of high-quality clinical research in the introduction and discussion sections of primary reports of clinical trials evaluating physiotherapy interventions, with only 41% of trial reports citing systematic reviews in the introduction and <1% numerically integrating trial results into the existing evidence in the discussion. Increasing these proportions would assist clinicians to use the trial results to guide clinical practice. Citation and integration could be increased by amending existing checklists and editorial guidelines. Items relevant to the citation of existing research in the introduction and discussion section could be made more explicit in future versions of the CONSORT statement⁹ because it is known to improve the completeness of reporting.¹⁰ The authors could be encouraged to cite relevant high-quality clinical research in their trial reports by making editorial guidelines more specific. For example, in 2005, The Lancet started asking authors to present the results of their trials within the context of previous research findings,³³ and from January 2015, the policy was strengthened to integrating their results into their own systematic review or into a systematic review published by others.³⁴

In addition to evaluating the time course of the increase in citation of systematic reviews in the introduction and discussion sections of trial reports, future research could also evaluate the utility of a interval-scale outcome measure to quantify citation, determine which factors are associated with citation of existing research and determine whether the citation and integration rates differ in other areas of health care. Some possible factors that may influence citation could be research funding (because of the rigorous review process) and journal impact factor (because high-impact journals are more likely to have more rigorous review procedures). To our knowledge, citation of systematic reviews in the introduction and discussion sections of reports of clinical trials has only been examined in medicine and physiotherapy. Evaluation of trials in other areas of health care is warranted.

Conclusion

Many clinical trials are reported without reference to the existing relevant high-quality research. Only 41% of reports of clinical trials of physiotherapy interventions cite a systematic review or other evidence in the introduction as part of the justification for the reported trial. Reports published in 2015 were 2.3 times more likely to cite a systematic review or other evidence compared to reports published in 2001. The integration of the new trial’s results into existing results with a meta-analysis in the discussion section was only performed in one trial report. The results of this study are similar to a survey of the same issue among medical trials, suggesting that this is a pervasive problem. Influencing factors for citation should be identified. Citation and integration could be increased by amending existing checklists and editorial guidelines.

Footnotes

Acknowledgements

We acknowledge the following people for extracting data from the non-English-language articles: Jooeun Song, Mandy Lau, Mareike Extra, Nina Wang, Nolwenn Poquet, Yen-Ning Lin, Tie Yamato, Bruno Saragiotto, Jean-Philippe Regnaux and Johnny Kang. We are also grateful to Prof. Dr Barbara Zimmermann for supervising Xenia Hoderlein’s bachelor’s thesis.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The Physiotherapy Evidence Database receives financial support from the Australian Physiotherapy Association (Foundation Partner); Motor Accident Insurance Commission, Transport Accident Commission and Chartered Society of Physiotherapy (Partners); and the physiotherapy organisations in 46 other countries.

References

Haynes

RB.

Clinical epidemiology how to do clinical practice research. 3rd ed. Philadelphia, PA: Lippincott Williams & Wilkins, 2006.

Clarke

Alderson

Chalmers

Discussion sections in reports of controlled trials published in general medical journals. JAMA 2002; 287: 2799–2801.

Cooper

ID.

How to write an original research paper (and get it published). J Med Libr Assoc 2015; 103: 67–68.

Cetin

Hackam

DJ.

An approach to the writing of a scientific manuscript. J Surg Res 2005; 128: 165–167.

Provenzale

JM.

Revising a manuscript: ten principles to guide success for publication. AJR Am J Roentgenol 2010; 195: W382–W387.

World Medical Association Inc. WMA Declaration of Helsinki: ethical principles for medical research involving human subjects, http://www.wma.net/en/30publications/10policies/b3/ (2013, accessed 15 June 2016).

Robinson

Goodman

SN.

A systematic examination of the citation of prior research in reports of randomized, controlled trials. Ann Intern Med 2011; 154: 50–55.

Castro

AA.

Structuring the discussion of scientific papers. Randomised controlled trial of structured discussions is needed. BMJ 1999; 319: 581.

Schulz

Altman

Moher

. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010; 340: c332.

10.

Turner

Shamseer

Altman

. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev 2012; 11: MR000030.

11.

Al-Shahi Salman

Beller

Kagan

. Increasing value and reducing waste in biomedical research regulation and management. Lancet 2014; 383: 176–185.

12.

Chalmers

Bracken

Djulbegovic

. How to increase value and reduce waste when research priorities are set. Lancet 2014; 383: 156–165.

13.

Glasziou

Altman

Bossuyt

. Reducing waste from incomplete or unusable reports of biomedical research. Lancet 2014; 383: 267–276.

14.

Ioannidis

Greenland

Hlatky

. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 2014; 383: 166–175.

15.

Chan

Song

Vickers

. Increasing value and reducing waste: addressing inaccessible research. Lancet 2014; 383: 257–266.

16.

Ferreira

Herbert

Crowther

. When is a further clinical trial justified? BMJ 2012; 345: e5913.

17.

Higgins

JPT

Green

. Cochrane handbook for systematic reviews of interventions version 5.1.0, March 2011, http://www.cochrane-handbook.org (accessed 13 July 2016).

18.

Clarke

Chalmers

Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents?

JAMA 1998; 280: 280–282.

19.

Clarke

Hopewell

Chalmers

Reports of clinical trials should begin and end with up-to-date systematic reviews of other relevant evidence: a status report. J R Soc Med 2007; 100: 187–190.

20.

Hoffmann

Glasziou

Boutron

. Better reporting of interventions: Template for Intervention Description and Replication (TIDieR) checklist and guide. BMJ 2016; 78: 175–188.

21.

Maher

Sherrington

Herbert

. Reliability of the PEDro scale for rating quality of randomized controlled trials. Phys Ther 2003; 83: 713–721.

22.

Yamato

Maher

Saragiotto

. How completely are physiotherapy interventions described in reports of randomised trials? Physiotherapy 2016; 102: 121–126.

23.

Moseley

Sherrington

Elkins

. Indexing of randomised controlled trials of physiotherapy interventions: a comparison of AMED, CENTRAL, CINAHL, EMBASE, hooked on evidence, PEDro, PsycINFO and PubMed. Physiotherapy 2009; 95: 151–156.

24.

Michaleff

Costa

Moseley

. CENTRAL, PEDro, PubMed, and EMBASE are the most comprehensive databases indexing randomized controlled trials of physical therapy interventions. Phys Ther 2011; 91: 190–197.

25.

Verhagen

De Vet

De Bie

. The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol 1998; 51: 1235–1241.

26.

Physiotherapy Evidence Database (PEDro). Frequently asked questions. 5: how trials are rated, http://www.pedro.org.au/english/faq/#question_five (2016, accessed 13 July 2016).

27.

Macedo

Elkins

Maher

. There was evidence of convergent and construct validity of Physiotherapy Evidence Database quality scale for physiotherapy trials. J Clin Epidemiol 2010; 63: 920–925.

28.

De Morton

. The PEDro scale is a valid measure of the methodological quality of clinical trials: a demographic study. Aust J Physiother 2009; 55: 129–133.

29.

Herbert

RD.

Confidence interval calculator, http://www.pedro.org.au/english/downloads/confidence-interval-calculator/ (2013, accessed 13 July 2016).

30.

Teasell

Bayona

Marshall

. A systematic review of the rehabilitation of moderate to severe acquired brain injuries. Brain Inj 2007; 21: 107–112.

31.

Zhang

Liu

. Acupuncture efficacy on ischemic stroke recovery: multicenter randomized controlled trial in China. Stroke 2015; 46: 1301–1306.

32.

Physiotherapy Evidence Database (PEDro). PEDro statistics, https://www.pedro.org.au/english/downloads/pedro-statistics/ (2017, accessed 17 January 2017).

33.

Kleinert

Benham

Collingridge

. Further emphasis on research in context. Lancet 2014; 384: 2176–2177.

34.

The Lancet . Information for authors, http://www.thelancet.com/pb/assets/raw/Lancet/authors/lancet-information-for-authors.pdf (2016, accessed 13 July 2016).