Searching for studies: A guide to information retrieval for Campbell systematic reviews

This guide is derived from the information in Chapter 4 of The Cochrane Handbook (Higgins et al., 2023; Lefebvre et al., 2023). Carol Lefebvre, Eric Manheimer and Julie Glanville kindly gave permission to the original Campbell Collaboration (CC) Information Retrieval Guide authors to use the chapter and chapter updates as the basis for this guide. In 2015 (Kugley et al., 2017) and most recently in 2023 the Campbell Information Retrieval Methods Group (CC-IRMG) revised this guide to reflect current CC areas of practice and recommendations in the Methodological Expectations of Campbell Collaboration Intervention Reviews (MECCIR), capture evolving practice and strategies for searching, and update links and descriptions of individual bibliographic and other resources.

This document outlines some general issues in searching for studies; describes the main sources of potential studies; and discusses how to plan the search process, design, and carry out search strategies, manage references found during the search process and document and report the search process. A list of abbreviations and definitions used in this guide can be found in Appendix V.

Who is this guide for?

This guide is meant for review authors and information specialists (IS) to reference in the planning and conduct of Campbell Systematic Reviews. The information in this guide is designed to assist authors wishing to undertake searches for studies and to provide background information so that they can better understand the search process.

This guide provides high-level, overview information on information retrieval principles and is not a substitute for the Help sections of individual databases or thesauri. Researchers who wish to search particular sources should familiarise themselves with the database before beginning a search.

Citation

MacDonald, H., Comer, C., Forster, M., Labelle, P., Marsalis, S., Nyhan, K., Premji, Z., Rogers, M., Splenda, R., Stansfield, C., & Young, S. (2024). Searching for Studies: A Guide to Information Retrieval for Campbell Review. Campbell Systematic Reviews, Issue, [e-page number]. [doi].

Role of the IS/librarian

The fundamental premise of this guide is that information retrieval is an essential component of the systematic review, analogous to the data collection phase of a primary research study. A thorough and unbiased compilation of all potentially relevant studies is one of the key characteristics of a systematic review. If the literature located is unrepresentative of the population of completed studies, the remainder of the review process will be compromised (Lefebvre et al., 2023; Rothstein & Hopewell, 2009).

Librarians and IS are experts in searching. They can play an integral role in the production of Campbell reviews. There is increasing evidence to support the involvement of an IS in systematic reviews (Spencer & Eldredge, 2018; Wang & Lin, 2022) and evidence that IS involvement improves the quality of various aspects of the search process and reporting (Aamodt et al., 2019; Meert et al., 2016; Metzendorf, 2016; Pawliuk et al., 2024; Ramirez et al., 2022; Rethlefsen et al., 2015; Schellinger et al., 2021; Wang & Lin, 2022).

IS and librarians can offer support to authors in study identification from the early planning stage to the final write-up of the review (Dalton, 2019; Foster, 2015; Ghezzi-Kopel et al., 2022; Spencer & Eldredge, 2018). They may be able to provide training in, or if they are a member of the research team perform, some or all of the following:

Selecting databases and other sources to search.

It is recommended that review authors seek guidance from an academic librarian or IS with experience in supporting systematic reviews. Researchers may want to consider inviting an IS or librarian to be part of their team to help ensure a robust search.

How to find an IS/librarian

Researchers at academic institutions can contact their institutional library to see if there is a librarian with experience in systematic reviews. Academic librarians may be able to offer support for systematic reviews or be part of a review team.

Summary points

If possible, invite an IS or librarian with experience in conducting searches for systematic reviews to collaborate as a co-author on your review.

General issues

This document is meant to provide general guidance to reviewers and to establish minimum standards for key information retrieval tasks. Although the guide speaks specifically to individuals planning to conduct a Campbell review, the policies, procedures (White, 2009), and guidelines are applicable to anyone interested in implementing information retrieval methods that maximise coverage and minimise bias. For a more comprehensive discussion of information retrieval for systematic reviews see White's (2009) chapter entitled Scientific Communication and Literature Retrieval (White, 2009) and other reports that have described searching challenges for systematic reviews of various topics (Aromataris & Riitano, 2014; Beahler et al., 2000; Blackhall & Ker, 2008; Ogilvie, 2005; Wu et al., 2012).

The basic requirements of the systematic review search are outlined in methodological and reporting guidelines. Familiarising yourself with these guidelines early and often are vital to establishing a high-quality search strategy for your review. For Campbell reviews, use the MECCIR standards (https://doi.org/10.17605/OSF.IO/KCSPX). The present document is also an important resource if developing a search strategy without an IS or librarian.

The nature of the social, behavioural, and educational sciences literature

Given the diverse nature of the research questions addressed in the social, behavioural, and educational sciences that encompass the CC's core areas of focus potentially relevant studies are likely to be widely distributed and unreliably categorised. While retrieval of information from the literature is a critical concern for any systematic reviewer, retrieval of information about complex social, behavioural, and educational interventions is likely to be especially challenging. Studies in the social sciences more often lack a structured abstract and strict adherence to common terminology compared to those of the medical sciences, resulting in the need for more sensitive, rather than specific, searches (Mallett et al., 2012; Petticrew, 2006, pp. 83–84). The review process also favours research produced in the Global North making finding studies published in the Global South challenging (Bol et al., 2023).

Studies versus reports of studies

Systematic reviews typically treat original studies as the unit of analysis. Individual studies may be reported in multiple publications or be associated with other studies (e.g., post hoc analyses or surveillance studies). Related publications may report unique information or be a source of unwanted duplicate data. Every possible effort should be made to flag associated or related publications for inspection by the review authors.

Types of studies

This guide focuses on searching for studies that evaluate the effectiveness of interventions. Although the general guidance in this document should be applicable to any review, more targeted guidance may be important for reviews that target a different study type. For example for cost effectiveness reviews, authors may seek guidance on searching from Chapter 7 in Evidence-Based Decisions and Economics (Glanville & Paisley, 2010) or methodologic reports (Alton et al., 2006; Glanville & Paisley, 2010; Glanville et al., 2009; McKinlay et al., 2006; Royle & Waugh, 2003; Sassi et al., 2002). When conducting an overview study or review of systematic reviews authors should consider other search outlets such as systematic review repositories and registries.

Copyright

It is Campbell policy that all review authors and others involved in the Collaboration should adhere to copyright legislation. With respect to searching for studies, this refers to adhering to the terms and conditions of use when searching databases and downloading records and adhering to copyright legislation when obtaining copies of articles. Review authors should seek guidance on this from their local academic librarian or copyright expert as copyright legislation varies across jurisdictions.

Summary points

CC review authors should, if possible, seek collaboration or advice from an academic librarian or IS with experience conducting searches for systematic reviews.

Subject databases

Subject databases are generally the best way to identify an initial set of relevant reports of studies within a specific field. The majority of field specific databases, such as ERIC (education) and APA PsycInfo (behavioural and mental health), include abstracts, and may include links to the full text of the scholarly literature. Access to full-text literature will vary based on open-access status and subscriptions tied to a user's institutional affiliations. These databases usually index journal and non-journal sources, as well as materials in languages beyond English. A key advantage of these databases is that they can be searched for keywords in the title or abstract and/or by using controlled vocabulary assigned to each record (see Section 6.4).

Decisions related to which subject-specific databases are to be searched will be influenced by the topic of the review, access to specific databases, and budget considerations. In addition to field-specific databases, there are also a variety of multi-disciplinary databases (see Appendix I) which can be worthwhile to search.

Most of the subject-specific databases are available on a subscription or ‘pay-as-you-go’ basis, with a few databases being freely available. Databases may be available at no cost to the individual through national providers, site-wide licences at institutions such as universities or hospitals, or through professional organisations as part of their membership packages. Access to databases is therefore likely to be limited to those databases that are available to members of the review team. Review authors should seek advice from their local librarian for access at their institution. For those review authors who do not have an affiliation with an institution or access to subscription databases, Appendix I offers a selection of free and low-cost databases.

A selection of the main subject-specific databases and multi-disciplinary databases are listed in Appendix I.

General databases

Other sources of study information

Handsearching key journals

Handsearching traditionally involves a manual page-by-page examination of the entire contents of a journal issue or conference proceedings to identify eligible studies. Electronic journals and online conference proceedings can also be ‘handsearched’, by scrolling through contents online. Like other supplementary search methods, handsearching might locate studies not indexed in bibliographic databases or that were missed by the database searches.

A Cochrane Methodology Review found that a combination of handsearching and electronic searching was necessary for full identification of relevant reports published in journals, even for those that are indexed in MEDLINE (Hopewell et al., 2007). A study in 2008 found that handsearching was useful for identifying trials reported in letters or abstracts (Richards, 2008). More recently, Cooper, Varley-Campbell, et al. (2018) found that handsearching had the best recall compared with other methods, but efficiency was poor.

As handsearching is a labour-intensive process, reviewers might consider prioritising journals in which significant numbers of included studies have been found through other search methods. This will ensure picking up studies that have not yet been indexed by the databases.

A note about predatory journals

Predatory journals are journals that claim to be legitimate research publications but misrepresent their publishing and editorial practices (Elmore & Weston, 2020). The quality of articles included in these types of journals can be questionable (Moher et al., 2017). Unfortunately, they can be found in some bibliographic databases (Boulos et al., 2022; Dadkhah et al., 2017) meaning they can end up in database search results. Section 6.5.6 provides some guidance on dealing with predatory journals.

Summary points

As there is no preferred single source of studies or trials that can be searched for Campbell reviews, a broad selection of databases and other sources needs to be searched to ensure a comprehensive search strategy.

Consult a librarian or IS to select national, regional, and subject-specific databases that index literature related to the topic of the review.

The CC review process

Before submitting a Campbell review, reviewers must first submit a title registration form that outlines the intended scope of the review. Once accepted, authors will submit a protocol to their CC group. Referees will review this protocol to ensure that all the necessary steps are to be completed correctly. This evaluation includes an extensive review of the search process used to identify studies. An information retrieval checklist (see Appendix IV) is used both at this stage and at the final review stage in the peer review process. It is recommended that reviewers consult this checklist to ensure their protocol matches the criteria that will be used to judge completeness.

Seed articles and search validation

Seed articles, also referred to as benchmark studies, are those articles that are known to meet the eligibility criteria of the review. They may be known to the review team at the outset or may be gathered via exploratory searching, citation harvesting from previous reviews on a similar topic, or contacting experts in the field. Each seed article should be evaluated against the eligibility criteria by the review team to ensure they meet criteria for inclusion and would pass, at a minimum, the title and abstract screening. However, seed articles that only partially meet the inclusion criteria may sometimes be included, especially where a search concept is known to be complex. In this case, the usefulness of these seed articles may be limited to only the relevant search concepts. Establishing a seed article set should be done as a precursor to the creation of a search strategy. The size of the seed article set depends on the breadth of the scope of the review. To be useful, it should be representative of the diversity of evidence expected to be included in the review This can include, but is not limited to, discipline, geographic region of the study, timeframe of publication, and variables pertaining to the review topic (such as study designs, independent and dependent variables).

The seed article set can be used during development of the search strategy and is also used as a test list for checking the performance of the search strategy. During search development, the seed set can be harvested for terms found in the title, abstract, and author-keyword fields, and for controlled vocabulary terms applied to these records in specific databases. After creation of the search, the seed articles can be tested for retrieval against the search strategy, to validate search logic, and characterise the performance of the search strategy. Ideally the test set should be retrieved in its entirety across the combined database search results. Reporting of the validation process used and the articles included in the test set is recommended either at the protocol stage or in the final review manuscript (Page, Moher, et al., 2021; Pullin et al., 2022).

Search updates

It is also recommended to check included studies against Retraction Watch (https://retractionwatch.com/) to ensure studies have not been retracted. For more information about how to deal with retracted publications see the Cochrane Handbook Technical Supplement to Chapter 4, Section 3.9 Identifying fraudulent studies, other retracted publications, errata and comments: further considerations (Lefebvre et al., 2023).

Consult Sections 9.2 and 9.3 for further information on the specific details to document and report in the final review manuscript.

An introduction

This section highlights some of the issues to consider when designing search strategies for databases within the social sciences, but it does not completely address the many complexities in this area. There are two important reasons for this:

Range of databases: Given the multidisciplinary nature of most social science research questions and the large selection of social science related databases, searches must be implemented in multiple databases. Terminology (both keywords and controlled vocabulary) will vary across these databases as different disciplines often use different words to describe the same thing.

Given the above, customised search strategies must be constructed for each database as terminology will vary across disciplines, and the way one searches will differ across databases. It is for these reasons that the construction and implementation of searches requires the skills of an IS or academic librarian as one risks missing retrieving key studies if searches are poorly constructed or improperly implemented.

The review inclusion criteria will inform how the search is designed. The inclusion criteria may specify the eligible study designs, participants, interventions, and outcomes. Other aspects to consider in planning a search include:

Geographic considerations

Sensitivity versus precision

Searches for systematic reviews aim to be as extensive as possible to ensure that as many as possible relevant studies are included in the review. It is, however, necessary to strike a balance between striving for comprehensiveness and maintaining relevance when developing a search strategy. Increasing the comprehensiveness (or sensitivity) of a search will reduce its precision and will retrieve more non-relevant articles.

Sensitivity is defined as the number of relevant reports identified divided by the total number of relevant reports in existence. Precision is defined as the number of relevant reports identified divided by the total number of reports identified. Developing a comprehensive search is an iterative process in which the terms that are initially used may be modified based on what has already been retrieved.

Search strategy structure and components

The structure of a search strategy should be based on the main concepts being examined in a review. For a Campbell review, the review title and stated objectives should provide these concepts. The eligibility criteria for studies to be included will further assist in the selection of appropriate subject headings and keywords for the search strategy. It is usually unnecessary and even undesirable, to search on every aspect of the review's research question. Although a question may address particular settings or outcomes, these concepts may not be well described in the title or abstract of an article and are often not well indexed with controlled vocabulary terms.

Generally speaking, a search strategy to identify intervention studies will typically include two concepts: (1) the condition of interest, that is, the population and (2) the intervention(s) evaluated. Sometimes a third concept may be included, the outcome(s), although as mentioned not all outcomes may be mentioned in the title or abstract. Search filters or limiting commands may be used to further narrow the results by study design, document type, dates, and so forth (see Section 6.5.6 Search filters vs. limiting commands).

Once decisions have been made regarding which databases will be searched, the following key decisions will need to be made:

What are the key concepts to be searched?

For systematic reviews, a single search strategy is designed to capture all relevant literature. Troubleshooting the search strategy is usually done in the main database(s) expected to provide the majority of relevant results.

Keywords and controlled vocabulary

Many databases can be searched in two ways. The first is to use keywords, or words which are found in database fields such as the title or abstract field (sometimes called ‘free text searching’ or ‘natural language terms’). The second is based on subject headings that are assigned to individual records when they are entered into the database. These terms may be described as ‘subject headings’, ‘thesaurus terms’, or ‘controlled vocabulary’. For systematic reviews it is recommended that both approaches are used in combination.

Most interfaces offer an option to browse the hierarchy of subject headings associated with a specific database. The list of subject headings will also include a search function, so keywords can be searched for within the hierarchy and mapped against the relevant heading. Usually, subject headings come with a scope note that provides a definition for the term. Headings can then be selected and added to the search. Subject headings may have a list of similar or related subject headings. These can be selected and used in the search strategy if they are relevant to a particular concept.

Before beginning the search, it is good practice to identify reviews that include similar concepts and examine their search strategies to see what controlled vocabulary they used. Similarly, records of studies that meet the inclusion criteria (i.e., seed articles, see Section 5.2) can be examined within the database. The list of subject headings and author assigned keywords should be available within the database record. Online tools such as the Yale MeSH Analyzer (https://mesh.med.yale.edu/) and PubReminer (https://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi) can help with this process.

Many databases offer useful features such as ‘Related Searches’ or ‘Find Similar Results’, or a list of subject headings extracted from the retrieved set will be displayed in a sidebar, along with the number of hits retrieved for each heading. The former is database specific and can be used to find related content and search terms based on the database indexing. The latter feature is especially useful as it may introduce new subject headings that were not previously considered while also providing an indication of the number of records within the database that contains a particular subject heading.

Identifying relevant keywords

Keywords can be identified through examining the titles and abstracts of relevant studies. It is essential that for any word selected, related terms and synonyms are also identified by using a thesaurus or dictionary. For example, for the concept of parental involvement the following terms could be considered: parental support, family support, family participation, and so forth. Review teams usually include an expert in the field who can also help identify synonyms that might have been overlooked. Checking search strategies in related systematic reviews is also a useful way of identifying more search terms.

It is important to remember that English words might be spelt differently across geographical regions, for example, color (US) and colour (UK). Both spellings should be used in a search strategy. Similarly, some words for the same thing vary by country. This is particularly true of drug names, for example, acetaminophen (US) and paracetamol (UK).

Reviewers should consider the individual database to be searched before choosing keywords. Databases which are topic-specific, for example, ERIC, APA PsycINFO or Ageline are useful because they are already limited in scope. It should not be necessary therefore to search for the term ‘education’ in ERIC, or ‘Older adults’ in Ageline or ‘Psychology’ in APA PsycINFO. Including these terms with other concepts may be overly limiting and will not be useful in databases that are already focused on these topics.

Text mining for term selection

Text mining is a technique that uses natural language processing, machine learning or statistical techniques (like term frequency analysis) to transform human language into something that a machine can interpret and analyse (Elliot et al., n.d.). Text mining can be used to assist in the identification of relevant text words for search strategy development and to improve the sensitivity and precision of searches. It should not completely replace the work of domain experts and the IS in identifying keywords but can be used to find keywords missed by the traditional term harvesting techniques described above and to generate a starting point for search strategy development.

Text mining in the context of term selection for systematic reviews typically involves the use of a set of relevant studies. These can be identified using targeted searches of a database. It can be useful to analyse text from relevant references that are not findable from free-text terms, such as a set of seed or benchmark studies identified by the research team, to inform discovery of additional search terms. Natural language processing or other text mining techniques are then applied to the titles, abstracts and/or full text of these studies. This should begin with the removal of ‘stop words’ (common words not likely to be meaningful keywords, like ‘and’, ‘or’ and ‘the’), and techniques to identify important terms such as term frequency analysis or keyword co-occurrence. Once identified, these keywords can be screened, and authors and IS can make informed decisions about a word's relevance to the search.

There are many tools and software available to facilitate the application of text mining for search strategy development, some of which are general text mining tools and some specifically developed for evidence synthesis. For more information about the use of text mining in systematic reviews and associated tools, see Section 3.2.3 of the Technical Supplement of Chapter 4 of the Cochrane Handbook (Lefebvre et al., 2023). However, some of these tools are only developed for use with references from healthcare databases, such as PubMed, whereas generic tools can be useful for combining references obtained from different sources. Lefebvre et al. (2023) discusses some of the ways text mining could be used for informing searches.

There is growing interest in the use of artificial intelligence tools such as ChatGPT to assist in systematic review processes. The value of these tools is just emerging and the evidence of their efficacy in designing a comprehensive search strategy is unclear (Alshami et al., 2023). In the meantime, ChatGPT or similar tools should be used cautiously.

Boolean operators (AND, OR and NOT)

A search strategy should build up the controlled vocabulary terms, keywords, synonyms, and related terms for each concept, joining together the terms within each concept with the Boolean ‘OR’ operator (see example search strategies in Section 6.5.8). This means that retrieved articles will contain at least one of these search terms. Search concepts should be developed for the population (or condition) and intervention(s). These should then be joined together with the ‘AND’ operator. When using multiple operators, it is important to nest the search terms. Nesting refers to the use of parentheses to organise a search statement (see Section 6.5.8).

This final step limits the retrieved set to articles that address all included concepts. A note of caution about this approach is warranted however: if an article does not contain at least one term from each of the three sets, it will not be retrieved. For example, if an index term has not been added to the record for the intervention and the intervention is not mentioned in the title and abstract, the article would be missed. A possible remedy is to omit one of the concepts and decide which records to check based on the number retrieved and the time available to check them.

The ‘NOT’ operator should be avoided where possible to avoid inadvertently removing relevant records from the search set. For example, searching for records indexed as (female NOT male) would remove any record that was about both males and females.

Phrase searching and proximity operators

Most databases offer the ability to search for two or more words, in the order specified, commonly by enclosing keywords within quotes. An example of phrase searching would be: ‘parental involvement’.

Many database providers allow the searcher to use proximity operators (e.g., NEAR, WITHIN, adj), sometimes called adjacency operators, that specifies the relationship of two or more terms within a field. This results in higher sensitivity than simple phrase searching but greater precision than use of the ‘AND’ operator. It is, therefore, desirable to use this operator where available and relevant.

For example, the proximity operator in EBSCO databases (N#) will find the search terms within a certain number of words of each other regardless of their order. So (parental N3 involvement) will find these two keywords within three words of each other regardless of the order in which they appear. Similarly, the proximity operator (W#) finds the words if they are within a certain number of words of one another, but in the order in which they have been entered. So (parent* W3 involvement) will find ‘parental involvement’ but not ‘involvement of the parents’.

The availability and commands of proximity operators will vary depending on the provider of the database. It is therefore important to consult the Help or Search Tips section within each database.

Truncation and wildcards

To be as comprehensive as possible, it is important to include a wide range of keywords for each of the concepts being searched, including relevant synonyms, related terms, and variant spellings. This might include the use of truncation and wildcards. Truncation, also called stemming, is a technique that broadens your search to include various word endings. The asterisk (*) symbol is commonly used for truncation, but some databases may use other symbols. For example, parent* would retrieve results containing the terms parent, parental, parenting, and so forth. Wildcards are used to represent one or more characters. Wildcards are commonly used to find words with variable spellings. For example, colo?r could retrieve results containing the terms color or colour. Refer to the help guide for each database to check the proper truncation and wildcard symbols to use in your search.

Field searching

Field searching involves searching specific database fields such as the title, abstract, or journal name. Field searching can be used instead of the default search or a combined field search in a database. For example, the Topic field in Web of Science includes title, abstract and author keywords. In CINAHL the default search includes Title, Abstract, and Subject Headings. Limiting to title, abstract and keyword fields is a more specific search and can reduce the number of results found. If your search terms are exhaustive within each concept, and those terms are not present in the title, abstract, or keywords, then it is unlikely that the article would be relevant to the scope of the review.

Language, date and document format restrictions

Review authors should justify the use of any restriction in the search strategy. Whenever possible, review authors should attempt to identify and assess for eligibility all possibly relevant reports irrespective of language of publication. However, translation of included studies at the data extraction stage can be time consuming and/or costly. Ideally no language restrictions should be included in the search strategy to minimise bias. The review team can use free online translation tools to translate titles and abstracts to determine eligibility for screening.

The application of a publication date restriction will depend on the research question being addressed. For example, if it is known that relevant studies would have been conducted only after a specific date (e.g., web-based learning in schools would not be addressed prior to the mid-1990s), a justifiable use of publication date limits can be applied during the search.

Any information about an eligible study may contain valuable details for analysis. Letters, comments, errata and preprints may all contain relevant information about a study (Iansavichene et al., 2008; Oikonomidi et al., 2020; Zeraatkar et al., 2022). However, some databases contain news and wire feeds which review teams may not wish to include in their search. Careful consideration should be given to document format restrictions.

Review teams should decide how they will deal with predatory journals. Identifying predatory journals at the search stage can be difficult as they can be found in bibliographic databases (Boulos et al., 2022; Dadkhah et al., 2017). Using a critical appraisal tool may help reduce the risk of including poor quality studies (Ross-White et al., 2019). Rice et al. (2021) suggest checking if open access journals are listed in the directory of open access journals (DOAJ) (https://doaj.org/) or conducting a sensitivity analysis with predatory papers excluded from the synthesis. Munn et al. (2021) provide guidance on dealing with predatory journals including checking if journals adhere to Committee on Publication Ethics (COPE) (https://publicationethics.org/) core practices, using the Think, Check, Submit checklist (https://thinkchecksubmit.org/) to determine journal credibility, or checking against a list of predatory journals.

Search filters versus limiting commands

Search filters, also called ‘search hedges’, are predefined search strategies that are designed to retrieve specific types of records, such as those of a particular methodological design, geographic location or topic. Limiting commands, on the other hand, are built-in database-specific filters that can be applied once a set of results has been generated or when the search is executed, depending on the database and platform. For example, in Web of Science, there is a Categories limiting command based on Web of Science journal subject groups, and ERIC has a Document type limiting command for different types of publications.

Used extensively in the medical and health sciences, search filters may also be used when searching in the social sciences, but with some caution. A search filters web site has been developed by the UK InterTASC Information Specialists Subgroup (ISSG), which is the group of information professionals supporting research groups within England and Scotland providing technology assessments to the National Institute for Health and Clinical Excellence (NICE) (Glanville et al., 2008). The purpose of the website is to list search filters and to point to critical appraisals of the various filters. The site includes, amongst others, methodological filters for identifying systematic reviews, randomised and non-randomised studies and qualitative research in a range of databases and across a range of service providers (https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/home).

However, there are two main challenges with using search filters in the social sciences. Firstly, databases in the social sciences tend not to be as thoroughly indexed as those in medicine and may use indexing inconsistently, if at all. Similarly, if a search filter that uses keywords is used, potentially relevant studies may be missed. This is especially true for methodology as it is often not found in social sciences abstracts. This may call for a broader approach to searching for methodological contents. Searching for specific study types along with general terms might be more useful.

TIP: When searching for methodological content include specific study types and general terms rather than relying solely on search filters.

Introduction

Database searches used to identify studies for systematic reviews tend to retrieve large numbers of references. To organise and work with these references, it is imperative that review teams manage them in an effective and efficient manner. Reference management software serves many purposes but can be used to store and organise references from databases and other sources, identify and remove duplicates, retrieve full text, tag references, and generate in-text citations along with formatted bibliographies. Of note, other tools described in Section 8.3 may also be used to accomplish similar tasks.

Reference management software

Specially designed bibliographic or reference management software such as EndNote (https://endnote.com/), Zotero (https://www.zotero.org/), or RefWorks (https://refworks.proquest.com/) is helpful and relatively easy to use to manage references found through searches, as well as the full text of studies. Search results may be imported directly into a central database or library, managed (elimination of duplicates, addition of notes, categorisation), and outputted in a variety of citation styles. In addition to the three managers mentioned above, other ones are listed in the Systematic Review Toolbox (http://systematicreviewtools.com/software.php). Some reference management tools are freely available while others require a subscription or a one-time fee. Some are desktop-based software that only work on certain operating systems while others are accessible via a web browser. These characteristics along with other features are compared on Wikipedia (https://en.wikipedia.org/wiki/Comparison_of_reference_management_software) and documented in the literature (Gibbs et al., 2022; Meade et al., 2024; Panda, 2023; Saxena & Kaushik, 2022).

When review teams have access to screening software or services such as those described in Section 8.3, reference management software may not be needed. However, there are instances where an RIS export from a database or resource might not be available and where a reference manager's browser extension might allow for certain bibliographic data to be captured. Another example might be when a search tool does not allow for searches to be combined and several separate searches need to be conducted. A reference manager may act as an intermediary to remove duplicates before importing those references into a screening tool. Some reference managers can also be used to find full-text articles in batches.

For review teams that do not have access to screening software or services, reference managers may also be used to review, screen, and select studies (Bramer et al., 2017; Mendes et al., 2019).

Removing duplicates

To identify studies, review teams will search multiple databases to ensure their search is comprehensive. Doing so will inevitably lead to finding the same reference more than once. It is imperative that review teams find an approach for removing duplicates efficiently to avoid extra work in screening the same reference more than once.

Screening tools described in Section 8.3 often remove duplicates when RIS files are imported. With reference managers, de-duplication features may vary. Some identify duplicates automatically while others require manual intervention.

De-duplication is fraught with potential issues and no single solution is foolproof. When automated, the process may be prone to identify false positives by removing references that are actually unique as well as false negatives by keeping references that are actually duplicates. This happens because bibliographic data for the same study may vary across databases, and algorithms used to identify duplicates may not capture these nuances. When done manually, identifying duplicates is extremely time-consuming.

Review teams may turn to published studies for guidance on best practices for removing duplicates such as considering free tools (Guimarães et al., 2022), comparing reference managers with screening tools (McKeown & Mir, 2021) or turning to newer de-duplication algorithms (Borissov et al., 2022). Because the number of duplicates removed across sources will need to be reported in the PRISMA flow diagram (Page, McKenzie, et al., 2021), it is imperative that this process be well thought out and documented.

Summary points

Manage references effectively through the review process by relying on reference management or screening software or services.

A typical process for selecting studies

A typical process for selecting studies may include the following steps.

Merging of records obtained from various database searches to remove duplicate records (identical records that have the same metadata such as title, authors, journal, year, etc.).

Multiple reports of the same study may exist. Authors should not discard multiple reports of the same study in case they contain additional details of the study design and conduct. Authors may merge these reports either before or after full-text evaluation. In other words, the unit of interest in a systematic review is a ‘study’ and not a ‘publication’ (of which there may be multiple for a given ‘study’).

Furthermore, authors may need to contact the corresponding author of reports of studies to confirm or clarify study details, if unclear in the available text, before making the final decisions.

The planned process, including specific references to process, persons involved, and software should be outlined in the protocol.

Implementation of the selection process

Once the search has been conducted and duplicate records removed, screening for eligible studies can commence. Decisions about eligibility will be made based on predetermined criteria for inclusion or exclusion established by the review authors during the protocol development stage. Screening usually happens in two steps: title/abstract screening followed by a review of full-text; however, teams may decide to conduct title-only screening as a first step prior to looking at abstracts.

Screening should be conducted with each record being reviewed by at least two independent reviewers (MECCIR Standard 4, https://doi.org/10.17605/OSF.IO/KCSPX) at each stage. It is highly recommended that the criteria be piloted on a set of studies. In this way, disagreements and misunderstandings about the criteria and concept definitions can be resolved before too many records have been processed. The aim is to have as clear criteria as possible to minimise the number of conflicting decisions between reviewers and ultimately reduce the risk of bias in record selection. Review teams typically develop additional guidance on applying the criteria, which can be updated for clarity and consistency through the screening process. Substantive changes that deviate from the protocol should be avoided throughout the review itself. Changes made during the review should be reported transparently in the final manuscript such that deviations from the protocol are clear.

The team may consider calculating the interrater reliability (IRR) of reviewers during the pilot and using a cutoff (e.g., 75% or 80%) (Aromataris et al., 2024; McHugh, 2012) to establish a transparent metric for concluding the pilot (Belur et al., 2021). This piloting can happen both at the title/abstract and full-text screening stages. For the title/abstract screening pilot, typically between 25 and 100 studies, randomly chosen from the results set, can be screened (Aromataris et al., 2024). More may need to be screened if the IRR is low. For the full-text screening pilot the number of studies is usually lower and will depend on how many studies are included at the full-test screening stage.

During title/abstract screening reviewers choose whether to move a study into the full-text screening phase. Only studies that can clearly be excluded based on the title and abstract alone are removed at this stage – this includes studies failing to meet one or more of the eligibility criteria. Studies that meet eligibility criteria or for which there is insufficient information in the title and abstract to decide are moved onto the full-text screening phase. When reviewers make conflicting decisions, these conflicts are resolved by a third reviewer or by discussion and consensus between reviewers. At the title/abstract stage, reasons for exclusion are not typically recorded. In mixed methods reviews, when screening for qualitative studies Skalidou and Oya (2018) describe a need to be flexible and inclusive to allow for variations in reporting.

The studies that move onto the full-text screening stage will be reviewed in full to make a final determination about their eligibility. Full-text articles can be found via library collections, open access journals or other sources where available, including interlibrary loan services provided by authors' institutional libraries. Studies for which the full text is not available after exhaustive attempts are noted as unretrievable. The total number of unretrieved records is reported in the PRISMA flow diagram (Page, McKenzie, et al., 2021). For retrieved articles, two independent reviewers make determinations about eligibility and as above, conflicts are resolved by a third reviewer or by discussion and consensus. For the full-text screening stage, reasons for exclusion are recorded and must also be agreed upon by reviewers. Reasons for exclusion should mirror the predefined eligibility criteria. Only one reason for exclusion can be recorded per article even if the article meets multiple exclusion criteria. It is therefore recommended that exclusion reasons be ordered (in the form of a hierarchy of exclusion reasons) based on importance or ease of determination, such that the first criteria not met by a record would be used and noted as the exclusion reason. It can be helpful to calculate and report the IRR (e.g., using Cohen's Kappa; simple percentage [number of records with agreement between reviewers/total number of records reviewed by those reviewers]) of the screening stages to help readers better understand the risk of bias in selection of included studies (Belur et al., 2021).

Software support for selecting studies

Because the searches in systematic reviews often result in thousands of records, the screening process can be labour-intensive and logistically challenging. For this reason, it is highly recommended that teams use specialised software for managing the screening process. These softwares can also facilitate the documentation of included and excluded studies, which is critical for proper reporting and the development of the PRISMA flow diagram. It is now common for these tools to incorporate machine learning and other tools/techniques that can increase speed and efficiency of the relevance screening phase.

Summary points

Merge records and remove duplicates.

Reporting guidelines

The search process needs to be documented in enough detail to ensure that it can be reported in the final manuscript such that all the searches of all databases are reproducible. It is helpful to become familiar with the requirements of relevant reporting guidelines at the start of the review to ensure the team documents all the required reporting items throughout the search process.

Documenting the search process

The information that your team documents and keeps track of during the search process should be informed by MECCIR and PRISMA-S reporting standards/guidelines (see Section 9.1). A well-documented search will facilitate writing this section of the review manuscript.

The search strategies for each database will need to be copied and pasted exactly as run and included in full, together with the total number of search results and the number of records downloaded. The number of records retrieved from each database search will need to be recorded in the Appendix. The search strategies should not be re-typed as this can introduce errors. Many databases offer an option to print, email, or download the search history which should be used where available. See Appendix III for a template that may be used for this purpose.

Summary points

Refer to the MECCIR standards, and ideally the PRSIMA-S reporting guideline early and often.