Abstract
Background
Systematic reviews are becoming increasingly important for policy-makers making decisions about health (Lavis et al., 2006; Petticrew et al., 2004; Welch et al., 2012). The World Health Organization (WHO) defines health as “a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity”(WHO, 2017). Systematic reviews offer many potential benefits to policy-makers, including identifying interventions that are effective or not effective, are considered to have lower risk of bias than other studies, and offer more confidence in results than single studies (Lavis et al., 2006a). However, health policies are often made without the use of research evidence (Oxman et al., 2009). Barriers to the use of research, specifically systematic reviews, in policymaking have been identified (Oliver et al., 2014). Most systematic reviews are written using technical language, are too long, and lack the contextual information important for policy-makers and other users making decisions about how to use the evidence in their decision making (Lavis et al., 2005a; Rosenbaum et al., 2011; Sheldon, 2005). Strategies to promote the use of research evidence to policy-makers are required, and evidence summaries have been suggested as a facilitator to evidence-informed decision making (Bunn & Sworn, 2011).
Systematic review summaries may be called evidence summaries, policy briefs, briefing papers, briefing notes, evidence briefs, abstracts, summary of findings, or plain language summaries (Adam et al., 2014). Within the Cochrane Collaboration, the Evidence Aid Project was developed in response to the 2004 Indian Ocean Tsunami as a means of providing decision makers and health practitioners ‘on the ground’ with summaries of the best available evidence needed to respond to emergencies and natural disasters (Kayabu & Clarke, 2013).
In addition to Evidence Aid, there are many organizations that develop and disseminate evidence summaries for different populations or subsets of decision makers. For example, SUPPORT Summaries were developed for policy-makers in low- and middle-income countries making decisions about maternal and child health programs and interventions (www.support-collaboration.org). Health Systems Evidence provides policy briefs for policy-makers making health systems decisions (www.healthsystemsevidence.org/). Communicate to vaccinate (COMMVAC) is creating user friendly summaries to translate evidence on vaccination communication for policy makers and the community in low- and middle-income countries (LMICs) (http://www.commvac.com). Rx for change is a searchable database for evidence about intervention strategies to alter behaviours of health technology prescribing, practice, and use (www.cadth.ca/resources/rx-for-change). Harvesting Evidence summarizes evidence on health systems and/or immunization for decision-making and implementation (http://www.harvesting-evidence.org). In fact, Lavis et al. identified 16 organizations involved in the production of summaries for policy makers in low- and middle-income countries (Adam et al., 2014).
Systematic review summaries consist of summarized evidence from systematic reviews intended to assist policy-makers in understanding the systematic review evidence and using it in their decision making. These interventions may include structured summaries (e.g. SUPPORT summaries, Evidence Aid), policy briefs which are based on systematic reviews (e.g. Health Systems Evidence), and plain language summaries, structured abstracts, and Summary of Findings tables (e.g. Cochrane reviews). These may be provided in print or web-based formats and are aimed at policy-makers, and other decision makers making decisions about health. The summaries may include information about the context in which the studies were conducted, the applicability of the results (e.g. SUPPORT Summaries comment on the relevance of the findings for disadvantaged communities), as well as the findings, methods and conclusions.
A needs assessment conducted by Evidence Aid found that systematic review summaries could improve understanding of users (i.e. non-governmental organizations (NGOs), health care providers) so that they can make decisions on the applicability of the findings to their local setting (Kayabu & Clarke, 2013). These user-friendly formats highlight the policy-relevant information and allow policy makers to quickly scan the document for relevance (Lavis et al., 2005a; Lavis et al., 2006a).
Evidence suggests that policy makers are more likely to use systematic reviews when the evidence is provided in a timely manner, and aligns with interests, values, and political goals of policymakers (Lavis et al., 2005a; Lavis et al., 2005b). Evidence summaries may increase the use of systematic review evidence by policymakers because they fulfil these by: 1) providing “user friendly” and plain language summaries of the evidence, 2) providing evidence “at-a-glance” with links to the complete systematic reviews, and 3) focusing on policy-relevant topics (Adam et al., 2014; Lavis et al., 2009). In addition, they may improve access to systematic reviews, because most organizations make summaries available freely through online databases and repositories (Adam et al., 2014).
Interest in the production and use of systematic review summaries is increasing, as evidenced by the growing number of organizations developing and disseminating them (Adam et al., 2014). However, evidence on the usefulness and effectiveness of systematic review derivatives is lacking. Previously conducted systematic reviews have looked at interventions to increase the use of systematic reviews among decision makers, however, these have focused on the use of complete systematic reviews in decision-making, and none identified focused specifically on derivatives of systematic reviews. For example, Murthy et al. conducted a systematic review examining the effectiveness of interventions for improving the use of systematic reviews in decision-making by health system managers, policy-makers, and clinicians (Murthy et al., 2012). Eight studies were included and the authors concluded that information provided as a single, clear message may improve evidence-based practice, but increasing awareness and knowledge of systematic review evidence might require a multi-faceted intervention. Similarly, Perrier et al. conducted a systematic review of interventions encouraging the use of systematic reviews by health policymakers and managers (Perrier et al., 2011). Four studies were included in the systematic review and the authors concluded that future research should identify how systematic reviews are accessed and the formats used to present the information. Finally, a review by Wallace et al. found that the facilitators to increase systematic review use by policymakers included description of benefits as well as harms and costs, and using a 1:3:25 staged approach to evidence summaries (Wallace, Byrne, & Clarke, 2012). However, none of these reviews were focused on summaries created from systematic reviews. In addition, we will focus on studies of evidence summaries for health policy-makers and health system managers making decisions on behalf of a large jurisdiction or organization but will not include studies related to decision making for an individual person or patient.
To the best of our knowledge, this is the first systematic review to assess the use of systematic review summaries in policy-making. The results of this review will inform researchers and systematic review summary developers of the best way to present the evidence to ensure that evidence summaries fulfill their goal of informing policy-makers with the best possible evidence needed to make health-related decisions.
OBJECTIVES
The objectives of this review are to 1) assess the effectiveness of evidence summaries on health policy-makers’ use of the evidence and 2) identify the most effective components of the summaries for increasing policy-makers’ use of the evidence.
METHODOLOGY
We will include studies examining the effects of any type of “friendly front end”, “evidence summary”, or “policy brief” or other product derived from systematic reviews or guidelines based on systematic reviews that presents evidence in a summarized form to health policy-makers and health system managers. We will exclude studies in which evidence summaries are one component of a multi-component intervention.
For this review, we define a systematic review as meeting the following criteria: clear inclusion and exclusion criteria, a comprehensive and systematic search strategy, explicit and reproducible methodology, an assessment of the validity of the findings of the included studies, and a systematic presentation and synthesis or meta-analysis (if possible) of the findings of the included studies (Campbell Collaboration, n.d.; Green et al., 2011). Interventions must include a summary of a systematic review and be actively “pushed” to target users, meaning that the summaries are packaged for decision makers and made accessible (Lavis et al., 2006a). For example, a potentially included study used an intervention that evaluated the effectiveness of friendly front ends by assessing changes in policy-maker beliefs (Beynon et al., 2012). An example of a study that would be excluded assessed the views of policymakers on how systematic reviews can be promoted within a low- and middle-income country (Yousefi-Nooraie et al., 2009).
We will include any comparisons including active comparators (e.g. other summary formats) or no intervention. This will allow us to assess both whether summaries increase health policy-makers’ use of the evidence when compared to no summaries or different types of summaries as well as whether some formats are more effective. We will assess studies using an active comparator separately from those with no intervention in the control group.
We will include studies which include health policy-makers at all levels (including: civil society organization staff, non-governmental organization staff, local government staff, regional government staff, federal government staff) and health system managers making decisions on behalf of a large jurisdiction or organization (Perrier et al., 2011). We will not include studies related to decision making for an individual person or patient. For the purposes of this review, we define ‘health policy-makers’ as those responsible for making decisions about healthcare policies and programs which are those intended to restore or maintain physical, mental, or emotional well being (WHO, 2017).
Primary outcomes
The primary outcomes are: Use of systematic review derivative product in decision making (e.g. self-reported use of the evidence in policy-making, decision-making as well as self-reported access of research, appraisal of research, or commissioning of further research within the decision-making process (Redman et al., 2015). We define “use” as instrumental, conceptual, or symbolic use of research in decision making. Instrumental use of is the direct use of research, conceptual use includes using research to gain an understanding of a problem or intervention, and symbolic use is the use of research to confirm a policy or program already implemented (Amara, Ouimet, & Landry, 2004). Understanding, knowledge, and/or beliefs (e.g. changes in knowledge scores about the topic included in the summary) as reported by the authors of the included studies.
Secondary outcomes
We will also include studies that report on any of the following outcomes: Perceived relevance of systematic review summaries Perceived credibility of the summaries Perceived usefulness and usability of systematic review summaries Perceptions and attitudes regarding the specific components of the summaries and their usefulness Understandability of summaries Desirability of summaries (e.g. layout, selection of images, etc)(Rosenbaum et al., 2011)
We recognize that some studies may use different terms to describe these outcomes. For example, the term ‘satisfaction’ maybe used as an umbrella term to capture relevance, usability, and desirability. These outcomes will be assessed by the team and categorized according to the above list.
We will include randomised controlled trials (RCTs), non-randomised controlled trials (NRCTs), controlled before-after CBA (studies), and interrupted time series (ITS) studies.
An Information Specialist will help develop the search strategy using the PRESS Guideline (Sampson et al., 2008). We will build on the search strategy used by Perrier et al. and Murthy et al. in their systematic reviews of interventions to encourage the use of systematic reviews by health managers and policy-makers.(Murthy et al., 2012; Perrier et al., 2011)
The search conducted by Perrier et al. identified 11,297 records (after removing duplicates) and included four papers reporting two studies. This search included the following databases: Medline, EMBASE, CINAHL, Cochrane Central Register of Controlled Trials, Cochrane Methodology Register, Health Technology Assessment Database, and LISA. We will modify these search strategies and expand this search by including additional databases, as suggested by John Eyres, of the International Initiative for Impact Evaluation (3ie) and the Campbell International Development Review Group. These include databases such as Global Health (CABI), Global Health Library (from WHO), Popline, Africa-wide, Public Affairs Information Service, Worldwide Political Science Abstracts, Web of Science, DfiD (Research for Development Database) (See Appendix 1 for MEDLINE search strategy).
We will search websites of research groups and organizations producing evidence summaries to identify unpublished studies evaluating the effectiveness of the systematic review derivatives in increasing policy-makers’ understanding (e.g. Health Systems Evidence, the Canadian Agencies for Health Technology Assessment, SUPPORT Summaries).
We will check reference lists of relevant studies to identify additional studies. We will contact researchers to identify ongoing and completed/published work. We will report the results of the search using the PRISMA flow diagram.
Two reviewers will independently screen titles and abstracts to identify relevant studies meeting the pre-specified inclusion criteria. The full text of potentially included studies will be screened independently by two authors. Data extraction and quality assessment will be conducted independently and in duplicate. We will use Covidence software (https://www.covidence.org/) for screening of studies. All completed studies will be included if they meet the inclusion criteria listed above.
The data extraction form will be pre-tested, and will include factors related to the population, intervention, comparison, and outcomes. The data will be extracted independently in duplicate by two reviewers using a structured Excel sheet and will be piloted on ten articles. Disagreements on extractions will be resolved by discussion and with a third member of the research team when necessary.
Data will be extracted for: Country Setting Study design Participants Type of policy or decision makers Country Age Gender Intervention Type of evidence summary Format of evidence summary Description of evidence summary components (e.g. descriptions of easy-to-skim formatting, graded entry, use of tables/figures)(Moat, Lavis, & Abelson, 2013) Mode of delivery Topic of evidence summary Recommendation of evidence summary Outcomes Policy/decision makers’ self-reported use of summaries in decision making Policy/decision makers’ knowledge of the summary content and the measurement used Policy/decision makers’ understanding and measurement used Perceived relevance of the summaries and measurement used Perceived credibility of the summaries and measurement used Perceived usefulness and usability of the summaries and measurement used Perceived understandability of the summaries and measurement used Perceived desirability of the summaries and measurement used Process Indicators How the systematic review was selected for summary (e.g. based on topic, quality criteria) How the evidence summary was developed (e.g. iterative process) Involvement of stakeholders in evidence summary development – which stakeholders, description of involvement
The methodological quality will be specifically examined using the risk of bias tools from the Cochrane Handbook for randomised trials and the Effective Practice and Organization of Care (EPOC) Review Group criteria for interrupted time series and controlled before after studies (Ballini et al., 2011; Higgins & Green, 2011) and the Risk of Bias in Non-randomised Studies – of Interventions (ROBINS-I) tool (Sterne et al., 2014) . Risk of bias will be assessed independently, in duplicate, by two authors and any discrepancies will be resolved by consensus and consultation with a third author, when necessary.
Effect estimates and confidence intervals for individual studies will be calculated (where possible) irrespective of whether a pooled effect estimate is calculated. When it is not possible to combine the data, we will present the results for each study separately.
When possible, any studies with cluster allocation (e.g. cluster-randomised trials, cluster-allocated controlled before and after studies, and interrupted time series) analyses with errors in the unit of analysis will be adjusted using the variance inflation factor, as described in the Cochrane Handbook, if the necessary data can be obtained from the study authors. We will obtain ICC from other similar studies with similar outcomes if the ICC is not published (eg by checking the Aberdeen website of ICCs, http://www.abdn.ac.uk/hsru/research/delivery/behaviour/methodological-research/, or the Campbell Collaboration website of ICCs for education). Sensitivity analyses will be used to assess the effects of incorporating these corrected analyses in our analysis.
We will attempt to contact the contact author of the studies by email for any missing data.
If more than 10 studies are included we will use funnel plots to explore publication bias.
Where appropriate, results will be synthesised using meta-analysis following the guidance outlined in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins & Green, 2011). When it is possible to combine studies, dichotomous outcomes will be reported as relative risks. Continuous outcomes will be reported as weighted mean differences. If an outcome has been reported in different scales (e.g. understanding), and we consider the scales to measure a similar construct, standardised mean differences will be used to summarise the data. We will analyse studies with an active comparator separately from those providing no intervention to the control group. Non-randomised studies will be meta-analyzed separately from RCTs. When results cannot be pooled, we will present a narrative summary of the results.
We will analyze the results of qualitative data from the included quantitative studies, when possible, to understand the perceptions and attitudes regarding the components of the summaries that were considered the most useful.
We will use the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to assess the quality of evidence for the outcomes reported in this review (Guyatt et al., 2011).
If meta-analysis is possible, we will explore heterogeneity using forest plots and the I2 statistic according to guidance of the Cochrane Handbook for Systematic Reviews of Interventions (Higgins & Green, 2011). Heterogeneity will be explored, if possible, by conducting meta-regression to assess the role of mediating factors, including: target audience of summary (e.g. focused on specific local context, generic summary) type of decision maker (e.g. federal policy-maker versus hospital administrator); and components of friendly front end (e.g. bulleted list, text, summary of findings table, causal chain).
Evidence summaries, like systematic reviews, that seek to inform decisions in a neutral way should not contain recommendations. Therefore, we will conduct a subgroup analysis to compare the effectiveness of summaries that provide recommendations and those without them.
The effect of including studies assessed as high risk of bias or studies in which there were unit of analysis errors that could not be re-analyzed will be considered in sensitivity analysis.
Footnotes
SOURCES OF SUPPORT
This work is funded by JP's CIHR Doctoral Research Award and a Campbell Systematic Review Award. The funders had no role in the development of this protocol.
DECLARATIONS OF INTEREST
The authors have no competing interests to declare.
REVIEW AUTHORS
The lead author is the person who develops and co-ordinates the review team, discusses and assigns roles for individual members of the review team, liaises with the editorial base and takes responsibility for the on-going updates of the review.
| Name: Jennifer Petkovic |
| Title: |
| Affiliation: Centre for Global Health, Bruyère Research Institute, University of Ottawa |
| Address: |
| 43 Bruyère Street, Annex E room 302 |
| City, State, Province or County: Ottawa, On |
| Postal Code: K1N 5C8 |
| Country: Canada |
|
|
| Name: Vivian Welch |
| Title: |
| Affiliation: Bruyère Research Institute, University of Ottawa, University of Ottawa, School of Epidemiology, Public Health and Preventive Medicine |
| Address: 43 Bruyère Street, Annex E room 304 |
| City, State, Province or County: Ottawa, On |
| Postal Code: K1N 5C8 |
| Country: Canada |
| Phone:613-562-6262 ext.2904 |
| Mobile: |
| Email: |
| Name: Peter Tugwell |
| Title: |
| Affiliation: University of Ottawa, Department of Medicine, Faculty of Medicine Ottawa Hospital Research Institute, Clinical Epidemiology Program University of Ottawa, Department of Epidemiology and Community Medicine, Faculty of Medicine, |
| Address: 43 Bruyère Street, Annex E room 302 |
| City, State, Province or County: Ottawa, On |
| Postal Code: K1N5C8 |
| Country: Canada |
| Phone:613-562-6262 ext. 2908 |
| Mobile: |
| Email: |
ROLES AND RESPONSIBLIITIES
Content: JP, VW, PT Systematic review methods: JP, VW Statistical analysis: JP, VW Information retrieval: APA, HC, MY
PRELIMINARY TIMEFRAME
Training and pilot testing on the inclusion criteria: Completion by May 20
Searches for eligible studies: Completion by May 25
Screening the results from the literature search: Completion by June 21
Training and pilot testing the study coding procedure: Completion by June 30
Extraction of data from eligible research reports: Completion by July 15
Statistical analysis: completion by July 31, 2016
Preparation of the final review report: completion by August 15, 2016
PLANS FOR UPDATING THE REVIEW
The review will be updated every two years. JP will be responsible for leading the updates.
AUTHORS’ RESPONSIBILITIES
By completing this form, you accept responsibility for preparing, maintaining and updating the review in accordance with Campbell Collaboration policy. The Campbell Collaboration will provide as much support as possible to assist with the preparation of the review.
A draft review must be submitted to the relevant Coordinating Group within two years of protocol publication. If drafts are not submitted before the agreed deadlines, or if we are unable to contact you for an extended period, the relevant Coordinating Group has the right to de-register the title or transfer the title to alternative authors. The Coordinating Group also has the right to de-register or transfer the title if it does not meet the standards of the Coordinating Group and/or the Campbell Collaboration.
You accept responsibility for maintaining the review in light of new evidence, comments and criticisms, and other developments, and updating the review at least once every five years, or, if requested, transferring responsibility for maintaining the review to others as agreed with the Coordinating Group.
PUBLICATION IN THE CAMPBELL LIBRARY
The support of the Campbell Collaboration and the relevant Coordinating Group in preparing your review is conditional upon your agreement to publish the protocol, finished review and subsequent updates in the Campbell Library. Concurrent publication in other journals is encouraged. However, a Campbell systematic review should be published either before, or at the same time as, its publication in other journals. Authors should not publish Campbell reviews in journals before they are ready for publication in the Campbell Library. Authors should remember to include a statement mentioning the published Campbell review in any non-Campbell publications of the review.
I understand the commitment required to undertake a Campbell review, and agree to publish in the Campbell Library. Signed on behalf of the authors:
Form completed by: Jennifer Petkovic
Date: 1 August 2017
Appendix 1: MEDLINE Search Strategy
1 ((systematic review$ or methodolog$ review$ or quantitativ$ review$ or qualitativ$ review$ or overview$ or synthes$ or metasynthes$ or megasynthes$) adj5 (decisionmak$ or decision-mak$ or policy-mak$ or policymak$ or policy decision$ or health$ polic$ or health$ manag$ or action$ or commission* or purchas* or procur* or budget hold* or budgethold* or service provi*or practice or application or implement$ or utili?ation or utili?ing or utili$ or disseminat$ or summar$ or hospital* decision* or treatment plan* or patient care or patient care or healthcare or health care or clinical decision* or pathway* or algorithm*)).ti,ab. (30531)
2 (((systematic adj2 (review* or overview* or synthesis or literature review* or evidence review*)) or methodolog* review* or quantitativ* review* or qualitative review* or overview or synthes* or metasynthes* or megasynthes*) adj5 (policy or policies or decision*)).ti. (400)
3 ((gap or gaps) adj7 ((knowledge or research or evidence or trial or result) adj2 practice)).ti,ab. (972)
4 1 or 2 or 3 (31697)
5 randomized controlled trial.pt. (409861)
6 controlled clinical trial.pt. (90286)
7 randomized.ab. (339312)
8 placebo.ab. (167495)
9 clinical trials as topic/ (175364)
10 randomly.ab. (244542)
11 trial.ti. (147363)
12 intervention*.ti. (96258)
13 or/5-12 (1074398)
14 State Medicine/ (52025)
15 exp Purchasing, Hospital/ (5754)
16 Contracts/ (2870)
17 exp Contract Services/ (12176)
18 exp Organizational Innovation/ (23682)
19 Insurance, Health/ or exp Managed Care Programs/ or Medicare/ (96098)
20 (commissioning or commissioner$).ti,ab. (4202)
21 (purchasing or purchaser$).ti,ab. (8743)
22 (procurement or procurer$).ti,ab. (6959)
23 (budget-holder$ or budgetholder$).ti,ab. (48)
24 (service adj2 (development or developer$ or provision or provider$)).ti,ab. (12067)
25 ((investment or budget or purchas$ or service) adj3 priorit$).ti,ab. (554)
26 priorit$ setting.ti,ab. (1563)
27 decision-maker$.ti,ab. (9028)
28 (contract$ adj3 (management or services or tender$)).ti,ab. (1293)
29 Decision Making, Organizational/ (10711)
30 exp Policy Making/ (20784)
31 exp Health Planning/ (297761)
32 or/14-31 (454534)
33 exp Evidence-Based Practice/ (68838)
34 Translational Research/ (6166)
35 exp “Diffusion of Innovation”/ (16708)
36 ((research or knowledge or innovation$ or evidence) adj5 (diffus$ or disseminat$ or implement$ or adoption or exchang$ or application or mobilis$ or mobiliz$ or synthes$ or transfer$ or translat$ or incorporat$ or uptak$ or utilis$ or utiliz$ or transmission or integrat$ or democratis$ or democratiz$ or shar$ or broke$)).ti,ab. (105758)
37 (‘research into practice’ or ‘knowledge into practice’ or ‘knowledge into action’ or ‘research into action’ or ‘research findings into action’ or ‘evidence into action’ or ‘evidence into practice’).ti,ab. (1564)
38 (KT adj5 (diffus$ or disseminat$ or implement$ or adoption or exchang$ or application or mobilis$ or mobiliz$ or synthes$ or transfer$ or translat$ or incorporat$ or uptak$ or utilis$ or utiliz$ or transmission or integrat$ or democratis$ or democratiz$ or shar$ or broke$)).ti,ab. (594)
39 ((evidence base$ or evidence inform$) adj5 (decision$ or plan$ or policy or policies or practice or action$)).ti,ab. (15612)
40 ((research or knowledge or innovation$ or evidence) adj5 (change$ or changing or improv$ or promot$ or influenc$ or impact$ or disinvest$ or discontinu$ or reject$ or abandon$ or ceas$ or restrict$ or disincentiv$ or stop$)).ti,ab. (116242)
41 ((research utiliz$ or research utilis$ or evidence or knowledge or innovation$) adj5 (decision-mak$ or decisionmak$ or policy-mak$ or policymak$ or health$ manag$ or health$ polic$ or action$ or practice or policy decision$)).ti,ab. (39062)
42 ((‘use’ or using or usage or useful or utiliz$ or utilis$) adj5 (evidence or research)).ti,ab. (83100)
43 Information Dissemination/ (11958)
44 (disseminat$ adj5 (findings or results)).ti,ab. (3346)
45 ‘Health Knowledge, Attitudes, Practice’/ (0)
46 Attitude of Health Personnel/ (98662)
47 Clinical Competence/ (71888)
48 or/34-47 (482218)
49 ((research or knowledge or innovation$ or evidence or information or policy) adj5 (brief$ or summar$ or synops$ or overview$ or bulletin$ or synthes$ or map or mapping or maps or framing$ or product$ or package$ or alert$ or commentar$ or strateg$ or algorithm$)).ti,ab. (116091)
50 (push activit* or pull activit*).ti,ab. (4)
51 (collaborat$ or ‘cross-profession$’ or intraprofession$ or intra-profession$ or interprofession$ or inter-profession$ or inter-disciplin$ or multi-disciplin$ or multi disciplin$ or multiprofession$ or outsourc$ or subcontract$).ti,ab. (106128)
52 ‘linkage.mp. and exchange’.ti,ab. [mp=title, abstract, original title, name of substance word, subject heading word, keyword heading word, protocol supplementary concept word, rare disease supplementary concept word, unique identifier] (1725)
53 or/48-51 (655194)
54 4 and 13 and (32 or 48 or 53) (867)
55 intervention?.ti. or (intervention? adj6 (clinician? or collaborat$ or community or complex or DESIGN$ or doctor? or educational or family doctor? or family physician? or family practitioner? or financial or GP or general practice? or hospital? or impact? or improv$ or individuali?e? or individuali?ing or interdisciplin$ or multicomponent or multi-component or multidisciplin$ or multi-disciplin$ or multifacet$ or multi-facet$ or multimodal$ or multi-modal$ or personali?e? or personali?ing or pharmacies or pharmacist? or pharmacy or physician? or practitioner? or prescrib$ or prescription? or primary care or professional$ or provider? or regulatory or regulatory or tailor$ or target$ or team$ or usual care)).ab. (195610)
56 (pre-intervention? or preintervention? or “pre intervention?” or post-intervention? or postintervention? or “post intervention?”).ti,ab. (13214)
57 demonstration project?.ti,ab. (2113)
58 (pre-post or “pre test$” or pretest$ or posttest$ or “post test$” or (pre adj5 post)).ti,ab. (78828)
59 (pre-workshop or post-workshop or (before adj3 workshop) or (after adj3 workshop)).ti,ab. (738)
60 trial.ti. or ((study adj3 aim?) or “our study”).ab. (766742)
61 (before adj10 (after or during)).ti,ab. (393848)
62 (“quasi-experiment$” or quasiexperiment$ or “quasi random$” or quasirandom$ or “quasi control$” or quasicontrol$ or ((quasi$ or experimental) adj3 (method$ or study or trial or design$))).ti,ab,hw. (115533)
63 (“time series” adj2 interrupt$).ti,ab,hw. (1414)
64 (time points adj3 (over or multiple or three or four or five or six or seven or eight or nine or ten or eleven or twelve or month$ or hour? or day? or “more than”)).ab. (11325)
65 pilot.ti. (47215)
66 Pilot projects/ (91909)
67 (clinical trial or controlled clinical trial or multicenter study).pt. (661665)
68 (multicentre or multicenter or multi-centre or multi-center).ti. (34249)
69 random$.ti,ab. or controlled.ti. (863821)
70 (control adj3 (area or cohort? or compare? or condition or design or group? or intervention? or participant? or study)).ab. not (controlled clinical trial or randomized controlled trial).pt. (470356)
71 evaluation studies as topic/ or prospective studies/ or retrospective studies/ (1072256)
72 (utili?ation or programme or programmes).ti. (60938)
73 (during adj5 period).ti,ab. (328074)
74 ((strategy or strategies) adj2 (improv$ or education$)).ti,ab. (22557)
75 “comment on”.cm. or review.pt. or (review not “peer review$”).ti. or randomized controlled trial.pt. (3239935)
76 (rat or rats or cow or cows or chicken? or horse or horses or mice or mouse or bovine or animal?).ti. (1429818)
77 exp animals/ not humans.sh. (4203554)
78 (or/55-74) not (or/75-77) (2709421)
79 4 and 78 and (32 or 48 or 53) (615)