Carol Lefebvre, Julie Glanville, Simon Briscoe, Robin Featherstone, Anne Littlewood, Chris Marshall, Maria-Inti Metzendorf, Anna Noel-Storr, Robin Paynter, Tamara Rader, James Thomas, L. Susan Wieland; on behalf of the Cochrane Information Retrieval Methods Group

Cite this chapter as: Lefebvre C, Glanville J, Briscoe S, Featherstone R, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Paynter R, Rader T, Thomas J, Wieland LS. Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane, 2022. Available from www.training.cochrane.org/handbook.

4.1 Introduction

Cochrane Reviews take a systematic and comprehensive approach to identifying studies that meet the eligibility criteria for the review. This chapter outlines some general issues in searching for studies; describes the main sources of potential studies; and discusses how to plan the search process, design and carry out search strategies, manage references found during the search process, correctly document the search process and select studies from the search results.

This chapter aims to provide review authors with background information on all aspects of searching for studies so that they can better understand the search process. All authors of systematic reviews should, however, identify an experienced medical/healthcare librarian or information specialist to provide support for the search process. The chapter also aims to provide advice and guidance for medical/healthcare librarians and information specialists (within and beyond Cochrane) involved in the search process to identify studies for inclusion in systematic reviews.

This chapter focuses on searching for randomized trials. Many of the search principles discussed, however, will also apply to other study designs. Considerations for searching for non-randomized studies are discussed in Chapter 24 (see also Chapter 19 when these are specifically for adverse effects). Other discussion of searching for specific types of evidence appears in chapters dedicated to these types of evidence, such as Chapter 17 on intervention complexity, Chapter 20 on economic evidence and Chapter 21 on qualitative evidence.

An online Technical Supplement to this chapter provides more detail on searching methods.

4.2 General issues

4.2.1 Role of the information specialist/librarian

Medical/healthcare librarians and information specialists have an integral role in the production of Cochrane Reviews. There is increasing evidence of the involvement of information specialists in systematic reviews (Spencer and Eldredge 2018) and evidence to support the improvement in the quality of various aspects of the search process (Rethlefsen et al 2015, Meert et al 2016, Metzendorf 2016, Aamodt et al 2019).

Most Cochrane Review Groups (CRGs) employ an information specialist to support authors. The range of services, however, offered by CRGs and/or their information specialists varies according to the resources available. Cochrane Review authors should, therefore, contact their Cochrane Information Specialist at the earliest stage to find out what advice and support is available to them. Authors conducting their own searches should seek advice from their Cochrane Information Specialist not only on which sources to search, but also with respect to the exact strategies to be run (see Section 4.4). If the CRG does not provide this service or employ an information specialist, we recommend that review authors seek guidance from a medical/healthcare librarian or information specialist, preferably one with experience in supporting systematic reviews.

Cochrane Information Specialists are responsible for providing assistance to authors with searching for studies for inclusion in their reviews, and for keeping up to date with Cochrane methodological developments in information retrieval (Cochrane Information Specialist Support Team 2021a). A key element of the role is the maintenance of a Specialized Register for their Review Group, containing reports of trials relating to the group’s scope. Within the limits of licensing restrictions, the content of these group registers is shared with users worldwide via the Cochrane Central Register of Controlled Trials (CENTRAL), part of the Cochrane Library (see Section 4.3.3).

Most CRGs offer support to authors in study identification from the early planning stage to the final write-up of the review, and the support available may include some or all of the following:

• advising authors on which databases and other sources to search;
• designing, or providing guidance on designing, search strategies for the main bibliographic databases and/or trials registers;
• running searches in databases and/or registers available to the information specialist;
• saving and collating search results, and sharing them with authors in appropriate formats;
• advising authors on how to run searches in other sources and how to download results;
• drafting, or assisting authors in drafting, the search methods sections of a Cochrane Protocol and Review and/or Update;
• ensuring that Cochrane Protocols, Reviews and Updates meet the requirements set out in the Methodological Expectations of Cochrane Intervention Reviews (MECIR) relating to searching activities for reviews;
• organizing translations, or at least data extraction, of study reports where required to enable authors to assess these reports for inclusion/exclusion in their reviews;
• obtaining copies of trial reports for review teams when required (within copyright legislation);
• providing advice and support to author teams on the use of reference management tools, and other software used in review production, including review production tools such as RevMan, Covidence and EPPI-Reviewer; and
• checking and formatting the references to included and/or excluded studies in line with the Cochrane Style Manual.

The Cochrane Information Specialists’ Handbook contains further information about how Cochrane Information Specialists can support authors (Cochrane Information Specialist Support Team 2021b).

4.2.2 Minimizing bias

Systematic reviews require a thorough, objective and reproducible search of a range of sources to identify as many eligible studies as possible (within resource limits). This is a major factor distinguishing systematic reviews from traditional narrative reviews, which helps to minimize bias and achieve more reliable estimates of effects and uncertainties. A search of MEDLINE alone is not considered adequate. Research evidence indicates that not all known published randomized trials are available in MEDLINE and that even if relevant records are in MEDLINE, it can be difficult to retrieve them (see Section 4.3.2).

Searching beyond MEDLINE is important not only for ensuring that as many relevant studies as possible are identified, but also to minimize selection bias for those that are found. Relying exclusively on a MEDLINE search may retrieve a set of reports unrepresentative of all reports that would have been identified through a wider or more extensive search of several sources.

Time and budget restraints require the review team to balance the thoroughness of the search with efficiency in the use of time and funds. The best way of achieving this balance is to be aware of, and try to minimize, the biases such as publication bias and language bias that can result from restricting searches in different ways (see Chapter 8 and Chapter 13 for further guidance on assessing these biases). Unlike for tasks such as study selection or data extraction, it is not considered necessary (or even desirable) for two people to conduct independent searches in parallel. It is strongly recommended, however, that all search strategies should be peer reviewed, before being run, by a suitably qualified and experienced medical/healthcare librarian or information specialist (see Section 4.4.8).

4.2.3 Studies versus reports of studies

Systematic reviews have studies as the primary units of interest and analysis. A single study may have more than one report about it (or record for it), and each of these reports or other records may contribute useful information for the review (see Section 4.6.1). For most of the sources listed in Section 4.3, the search process will retrieve individual reports of studies, so that multiple reports of the same study will need to be identified and associated with each other manually by the review authors. There is, however, an increasing number of study-based sources, which link multiple records of the same study together, such as the Cochrane Register of Studies and the Specialized Registers of a number of CRGs (see online Technical Supplement), and some other trials registers and regulatory and industry sources. Processes and software to select and group publications by study are discussed in Section 4.6.

4.2.4 Copyright and licensing

All review authors and others involved in Cochrane should adhere to copyright legislation and the terms of database licensing agreements. With respect to searching for studies, this refers in particular to adhering to the terms and conditions of use when searching databases and other sources and downloading records, as well as adhering to copyright legislation when obtaining copies of publications. Review authors should seek guidance on this from their medical/healthcare librarian or information specialist, as copyright legislation varies across jurisdictions and licensing agreements vary across organizations.

4.3 Sources to search

4.3.1 Bibliographic databases

4.3.1.1 Introduction to bibliographic databases

The search for studies in a Cochrane Review should be as extensive as possible in order to reduce the risk of reporting bias and to identify as much relevant evidence as possible (see MECIR Box 4.3.a). Searches of health-related bibliographic databases are generally the most efficient way to identify an initial set of relevant reports of studies (EUnetHTA JA3WP6B2-2 Authoring Team 2019). Database selection should be guided by the review topic (Suarez-Almazor et al 2000, Stevinson and Lawlor 2004, Lorenzetti et al 2014). When topics are specialized, cross-disciplinary, or involve emerging technologies (Rice et al 2016), additional databases may need to be identified and searched (Wallace et al 1997, Stevinson and Lawlor 2004, Frandsen et al 2019a).

MECIR Box 4.3.a Relevant expectations for conduct of intervention reviews

The three bibliographic databases generally considered to be the most important sources to search for reports of trials are CENTRAL (Noel-Storr et al 2020), MEDLINE (Halladay et al 2015, Sampson et al 2016) and Embase (Woods and Trewheellar 1998, Sampson et al 2003, Bai et al 2007). These databases are described in more detail in Sections 4.3.1.2 and 4.3.1.3 and in the online Technical Supplement. For Cochrane Reviews, CENTRAL, MEDLINE and Embase (if access to Embase is available to the review team) should be searched (see MECIR Box 4.3.a). These searches may be undertaken specifically for the review, or indirectly by searching the CRG’s Specialized Register.

Some bibliographic databases, such as MEDLINE and Embase, include abstracts for the majority of recent records. A key advantage of such databases is that they can be searched electronically both for words in the title or abstract and by using the standardized indexing terms, or controlled vocabulary, assigned to each record (see Section 4.3.1.2). Cochrane has developed a database of reports of randomized trials called the Cochrane Central Register of Controlled Trials (CENTRAL), which is published within the Cochrane Library (see Section 4.3.1.3).

Bibliographic databases are available to individuals for a fee (by subscription or on a ‘pay-as-you-go’ basis) or free at the point of use. They may be available through national provisions, site-wide licences at institutions such as universities or hospitals, through professional organizations as part of their membership packages or free-of-charge on the internet. Some international initiatives provide free or low-cost online access to databases (and full-text journals) over the internet. The Health InterNetwork Access to Research Initiative (HINARI) programme, set up by the World Health Organization (WHO) together with major publishers, provides access to a wide range of databases including the Cochrane Library for healthcare professionals in local, not-for-profit institutions in more than 120 countries, areas and territories. The International Network for the Availability of Scientific Publications (INASP) also provides access to a wide range of databases (and journals) including the Cochrane Library. Electronic Information for Libraries (EIFL) is a similar initiative based on library consortia to support affordable licensing of journals and other sources in more than 50 low-income and transition countries in central, eastern and south-east Europe, the former Soviet Union, Africa, the Middle East and South-east Asia.

The online Technical Supplement provides more detailed information about how to search these sources and other databases. The accompanying Appendix provides a list of general healthcare databases by region and healthcare databases by subject area. Further evidence-based information about sources to search can be found on the SuRe Info portal, which is updated twice per year.

4.3.1.2 MEDLINE and Embase

Cochrane Reviews of interventions should include a search of MEDLINE (see MECIR Box 4.3.a). MEDLINE (as of January 2022) contains approximately 30 million references to journal articles in biomedicine and health from 1946 onwards. More than 5000 journals in about 40 languages are indexed for MEDLINE (US National Library of Medicine 2021).

PubMed provides access to a free version of MEDLINE that also includes up-to-date citations not yet indexed for MEDLINE (US National Library of Medicine no date). Additionally, PubMed includes records from journals that are not indexed for MEDLINE and records considered ‘out-of-scope’ from journals that are partially indexed for MEDLINE (US National Library of Medicine 2020). Further details about MEDLINE, PubMed and PubMed Central and how they differ are available (US National Library of Medicine 2020).

MEDLINE is also available on subscription from a number of other database vendors, such as EBSCO, Ovid, ProQuest and STN. Access is usually ‘free at-the-point-of-use’ to members of the institutions paying the subscriptions (e.g. hospitals and universities). Ovid MEDLINE (segment name ‘medall’) covers all of the available content and metadata in PubMed with a delay of one working day (except during the annual reload, at the end of each year, when Ovid MEDLINE will not match the PubMed baseline). Aside from the MEDLINE records, Ovid includes all content types available in PubMed including; Epub Ahead of Print, PubMed-not-MEDLINE, In-process citations and citations for books available on the NCBI Bookshelf.

When searching MEDLINE via service providers or interfaces other than Ovid or PubMed, we recommend verification of the exact coverage of the database in relation to PubMed, where no explicit information on this is readily available.

Cochrane Reviews of interventions should include a search of Embase (if access to Embase is available to the review team) (see MECIR Box 4.3.a). Embase (as of January 2022) contains over 35 million records from 1974 onwards, including records from more than 8000 currently published journals from approximately 100 countries (Elsevier 2022). Embase now includes all MEDLINE records, thus, technically, allowing both databases to be searched simultaneously. Further details on the implications of this for searching are available in the online Technical Supplement. There are more than 10 million records in Embase from approximately 3000 journals that are not indexed in MEDLINE (Elsevier 2022, Ziel 2022). Embase Classic provides access to almost two million records digitized from the Excerpta Medica print journals (the original print indexes from which Embase was created) from 1947 to 1973 (Elsevier 2020). Embase now also includes pre-print articles from multiple sources (Elsevier 2021).

Embase is only available by subscription, either directly via Elsevier (as Embase.com) or from other database vendors, such as Ovid, ProQuest or STN. It is mandatory for Cochrane intervention reviews to include a search of Embase if access is available to the review team (see MECIR Box 4.3.a). Note that Embase is searched regularly by Cochrane for reports of trials. These records are included in CENTRAL (see online Technical Supplement).

The online Technical Supplement provides guidance on how to search MEDLINE and Embase for reports of trials. The actual degree of reference overlap between MEDLINE and Embase varies widely according to the topic, but studies comparing searches of the two databases have generally concluded that a comprehensive search requires that both databases be searched (Lefebvre et al 2008, Bramer et al 2016) (see MECIR Box 4.3.a).

Conversely, two studies examined different samples of Cochrane Reviews and identified the databases from which the included studies of these reviews originated (Halladay et al 2015, Hartling et al 2016). Halladay showed that the majority of included studies could be identified via PubMed (range 75% to 92%) and Hartling showed that the majority of included studies could be identified by using a combination of two databases, but the two databases were different in each case. Both studies, one across all healthcare areas (Halladay et al 2015) and the other on child health (Hartling et al 2016), report a minimal extent to which the inclusion of studies not indexed in PubMed altered the meta-analyses. PubMed coverage across systematic review topics has been further evaluated in a recent study based on a comprehensive sample of Cochrane Reviews. It provides further evidence of PubMed’s generally high coverage (range 68% to 73%), with an emphasis that it is markedly variable across and within specialties (Frandsen et al 2019b, Metzendorf and Featherstone 2019). Hence, the current recommendation of searching multiple databases needs to be evaluated further, so as to confirm under which circumstances comprehensive searches of multiple databases are warranted.

4.3.1.3 The Cochrane Central Register of Controlled Trials (CENTRAL)

Since its inception, the Cochrane Central Register of Controlled Trials (CENTRAL) has been recognized as the most comprehensive source of reports of randomized trials (Egger and Smith 1998). A more recent study by Noel-Storr et al reconfirmed the high sensitivity of CENTRAL in identifying randomized controlled trials (Noel-Storr et al 2020). CENTRAL is published as part of the Cochrane Library and is updated monthly. As of January 2022, CENTRAL contains over 1,800,000 records of reports of trials/trials registry records potentially eligible for inclusion in Cochrane Reviews, by far the majority of which are randomized trials (Noel-Storr et al 2020).

Many of the records in CENTRAL have been identified through systematic searches of MEDLINE, Embase, CINAHL Plus, Australian Index Medicus, KoreaMed, ClinicalTrials.gov and the trial records available through the WHO International Clinical Trials Registry Portal (see online Technical Supplement). CENTRAL, however, also includes citations to reports of randomized trials that are not indexed in MEDLINE, Embase or other bibliographic databases; citations published in many languages; and citations that are available only in conference proceedings or other sources that are difficult to access. It also includes records from trials registers and trials results registers beyond ClinicalTrials.gov and the WHO portal.

These additional records are, for the most part, identified by Cochrane Information Specialists, many of whom conduct comprehensive searches to populate CRG Specialized Registers, collecting records of trials eligible for Cochrane Reviews in their field. These Specialized Registers are included in CENTRAL. Where a Specialized Register is available, for which sufficiently comprehensive searching has been conducted, a search of the Specialized Register may be conducted instead of separately searching CENTRAL, MEDLINE and Embase for a specific review. In these cases, the search will be more precise, but an equivalent number of included studies will be identified with lower numbers of records to screen. There will, however, be a time-lag between records appearing in databases such as MEDLINE or Embase and their inclusion in a Specialized Register.

CENTRAL is available through the Cochrane Library. Many review authors have full access free-of-charge at the point-of-use through national provisions and other similar arrangements, or as part of a paid subscription to the Cochrane Library. All Cochrane Information Specialists have full access to CENTRAL.

The online Technical Supplement provides information on what is in CENTRAL from MEDLINE, Embase and other sources, as well as guidance on searching CENTRAL.

4.3.1.4 Other bibliographic databases

Many countries and regions produce bibliographic databases that focus on the literature produced in those regions and which often include journals and other literature not indexed elsewhere. There are also subject-specific bibliographic databases, such as AMED (alternative and allied therapies), CINAHL (nursing and allied health) and PsycINFO (psychology and psychiatry). It is highly desirable that searches be conducted of appropriate national, regional and subject specific bibliographic databases (see MECIR Box 4.3.b). Further details are provided in the online Technical Supplement.

Citation indexes are bibliographic databases that record instances where a particular reference is cited, in addition to the standard bibliographic content. Citation indexes can be used to identify studies that are similar to a study report of interest, as it is probable that other reports citing or cited by a study will contain similar or related content. Further details are provided in the online Technical Supplement.

MECIR Box 4.3.b Relevant expectations for conduct of intervention reviews

4.3.2 Ongoing studies and unpublished data sources

Initiatives to provide access to ongoing studies and unpublished data constitute a fast-moving field (Isojarvi et al 2018). Review authors should therefore consult their medical/healthcare librarian or information specialist for current advice.

It is important to identify ongoing studies, so that when a review is updated these can be assessed for possible inclusion. Awareness of the existence of a possibly relevant ongoing study and its expected completion date might affect not only decisions with respect to when to update a specific review, but also when to aim to complete a review. Information about possibly relevant ongoing studies should be included in the review in the ‘Characteristics of ongoing studies’ table.

Even when studies are completed, some are never published. An association between ‘statistically significant’ results and publication has been documented across a number of studies, as summarized in Chapter 13. Finding out about unpublished studies, and including their results in a systematic review when eligible and appropriate (Cook et al 1993), is important for minimizing bias. Several studies and other articles addressing issues around identifying unpublished studies have been published (Easterbrook et al 1991, Weber et al 1998, Manheimer and Anderson 2002, MacLean et al 2003, Lee et al 2008, Chan 2012, Bero 2013, Schroll et al 2013, Chapman et al 2014, Kreis et al 2014, Scherer et al 2015, Hwang et al 2016, Lampert et al 2016).

There is no easy and reliable single way to obtain information about studies that have been completed but never published. There have, however, been several important initiatives resulting in better access to studies and their results from sources other than the main bibliographic databases and journals. These include trials registers and trials results registers (see Section 4.3.3), regulatory agency sources and clinical study reports (CSRs); (the very detailed reports prepared by industry for regulatory approval) (see Section 4.3.4). A recent study (Halfpenny et al 2016) assessed the value and usability for systematic reviews and network meta-analyses of data from trials registers, CSRs and regulatory authorities, and concluded that data from these sources have the potential to influence systematic review results. Two earlier studies showed that a considerably higher proportion of CSRs prepared for regulatory approval of drugs provided complete information on study methods and results than did trials register records or journal publications (Wieseler et al 2012) and that conventional, publicly available sources (European Public Assessment Reports, journal publications, and trials register records) provide insufficient information on new drugs, especially on patient relevant outcomes in approved subpopulations (Köhler et al 2015).

A Cochrane Methodology Review examined studies assessing methods for obtaining unpublished data and concluded that those carrying out systematic reviews should continue to contact authors for missing data and that email contact was more successful than other methods (Young and Hopewell 2011). An annotated bibliography of published studies addressing searching for unpublished studies and obtaining access to unpublished data is also available (Arber et al 2013). One particular study focused on the contribution of unpublished studies, including dissertations, and studies in languages other than English, to the results of meta-analyses in reviews relevant to children (Hartling et al 2017). They found that, in their sample, unpublished studies and studies in languages other than English rarely had any impact on the results and conclusions of the review. They did, however, concede that inclusion of these study types may have an impact in situations where there are few relevant studies, or where there are ‘questionable vested interests’ in the published literature.

Correspondence can be an important source of information about unpublished studies. It is highly desirable for authors of Cochrane Reviews of interventions to contact relevant individuals and organizations for information about unpublished or ongoing studies (see MECIR Box 4.3.c). Letters of request for information can be used to identify completed but unpublished studies. One way of doing this is to send a comprehensive list of relevant articles along with the eligibility criteria for the review to the first author of reports of included studies, asking if they know of any additional studies (ongoing or completed; published or unpublished) that might be relevant. This approach may be especially useful in areas where there are few trials or a limited number of active research groups. It may also be desirable to send the same letter to other experts and pharmaceutical companies or others with an interest in the area. Some review teams set up websites for systematic review projects, listing the studies identified to date and inviting submission of information on studies not already listed. A recent study assessed the value of contacting trial authors and concluded that data supplied by authors modified the outcomes of some systematic reviews, but this was poorly reported in the reviews (Meursinge Reynders et al 2019). A recent case study evaluated the effectiveness, efficiency, cost and value of contacting study authors in a systematic review and concluded that this was cost-effective in terms of time taken and costs in carrying out this work compared with unique data identified from the authors’ replies (Cooper et al 2019). Another case study of a Cochrane Methodology Review reported that making contact with clinical trials units and trial methodologists provided data for six of the 38 RCTs included in the review, which had not been identified through other search methods (Brueton et al 2017).

MECIR Box 4.3.c Relevant expectations for conduct of intervention reviews

Asking researchers for information about completed but never published studies has not always been found to be fruitful (Hetherington et al 1989, Horton 1997) though some researchers have reported that this is an important method for retrieving studies for systematic reviews (Royle and Milne 2003, Greenhalgh and Peacock 2005, Reveiz et al 2006). A recent study reported successful outcomes of a digital media strategy to obtain unpublished data from trial authors (Godard-Sebillotte et al 2018). A study assessed the value of requesting information from drug manufacturers for systematic reviews and concluded that this helped to reduce reporting and publication bias and helped to fill important gaps, sometimes leading to new or altered conclusions, primarily where no other evidence existed (McDonagh et al 2018). The RIAT (Restoring Invisible and Abandoned Trials) initiative (Doshi et al 2013) aims to address the problems outlined above by offering a methodology that allows others to re-publish mis-reported and to publish unreported trials. Anyone who can access the trial data and document trial abandonment can use this methodology. The RIAT Support Centre offers free-of-charge support and competitive funding to researchers interested in this approach. It has also been suggested that legislation such as Freedom of Information Acts in various countries might be used to gain access to information about unpublished trials (Bennett and Jull 2003, MacLean et al 2003).

4.3.3 Trials registers and trials results registers

Cochrane Reviews of interventions should search relevant trials registers and repositories of results (see MECIR Box 4.3.d). A recent study suggested that trials registers are an important source for identifying additional randomized trials (Baudard et al 2017). A recent audit by Cochrane investigators showed that the majority of Cochrane Reviews do comply with this standard (Berber et al 2019). Although there are many other trials registers, ClinicalTrials.gov and the WHO International Clinical Trials Registry Platform (ICTRP) portal (Pansieri et al 2017) are considered to be the most important for searching to identify studies for a systematic review. Research has shown that even though ClinicalTrials.gov is included in the WHO ICTRP Search Portal, not all ClinicalTrials.gov records could be successfully retrieved via searches of the ICTRP Search Portal (Glanville et al 2014, Knelangen et al 2018). The extent to which this might still be the case with the new ICTRP interface released in its final version in June 2021 (see online Technical Supplement) remains to be ascertained. Therefore, the current guidance that it is not sufficient to search the ICTRP alone still stands, pending further research. A recent study reviewed the search interfaces of the EU Clinical Trials Register (EUCTR), ClinicalTrials.gov and the WHO ICTRP and offers further insights into how to search these resources (Cooper et al 2021a). Guidance for searching these and other trials registers is provided in the online Technical Supplement.

In addition to Cochrane, other organizations also advocate searching trials registers. These include the Agency for Healthcare Research and Quality (AHRQ) in the US, the European Network for Health Technology Assessment (EUnetHTA), the Institute for Quality and Efficiency in Health Care (IQWiG) in Germany, the Institute of Medicine in the US, and the National Institute of Health and Care Excellence (NICE) in the UK (Institute of Medicine 2011, Agency for Healthcare Research and Quality 2014, EUnetHTA JA3WP6B2-2 Authoring Team 2019, Institute for Quality and Efficiency in Health Care 2020, National Institute for Health and Care Excellence 2020).

There has been an increasing acceptance by investigators of the importance of registering trials at inception and providing access to their trials results. Despite perceptions and even assertions to the contrary, however, there is no global, universal legal requirement to register clinical trials at inception or at any other stage in the process, although some countries are beginning to introduce such legislation (Viergever and Li 2015).

Efforts have been made by a number of organizations, including organizations representing the pharmaceutical industry and individual pharmaceutical companies, to begin to provide central access to ongoing trials and in some cases trial results on completion, either on a national or international basis. A recent audit of pharmaceutical companies’ policies on access to trial data, results and methods, however, showed that the commitments made by companies to transparency of trials were highly variable (Goldacre et al 2017). Increasingly, as already noted, trials registers such as ClinicalTrials.gov also contain the results of completed trials, not just simply listing the details of the trial.

MECIR Box 4.3.d Relevant expectations for conduct of intervention reviews

4.3.4 Regulatory agency sources and clinical study reports

A number of organizations, including Cochrane, recommend searching regulatory agency sources and clinical study reports. These include the Agency for Healthcare Research and Quality (AHRQ) in the US, the Institute for Quality and Efficiency in Health Care (IQWiG) in Germany, and the Institute of Medicine in the US (Institute of Medicine 2011, Agency for Healthcare Research and Quality 2014, Institute for Quality and Efficiency in Health Care 2020). Potentially relevant regulatory agency sources include the EU Clinical Trials Register, Drugs@FDA and OpenTrialsFDA. Details of these are provided in the online Technical Supplement. Clinical study reports (CSRs) are the reports of clinical trials providing detailed information on the methods and results of clinical trials submitted in support of marketing authorization applications. In late 2010, the European Medicines Agency (EMA) began releasing CSRs (on request) under their Policy 0043. In October 2016, they began to release CSRs under their Policy 0070. The policy applies only to documents received since 1 January 2015. The terms of use for access are based on the purposes to which the clinical data will be put. Further details of this and other resources are available in the online Technical Supplement.

A recent study by Jefferson and colleagues (Jefferson et al 2018) that looked at use of regulatory documents in Cochrane Reviews, found that understanding within the Cochrane community was limited and guidance and support would be required if review authors were to engage with regulatory documents as a source of evidence. Specifically, guidance on how to use data from regulatory sources is needed. For more information about using CSRs, see the online Technical Supplement. Further guidance on collecting data from CSRs is provided in Chapter 5, Section 5.5.6.

4.3.5 Other sources

The online Technical Supplement describes several other important sources of reports of studies. The term ‘grey literature’ is often used to refer to reports published outside of traditional commercial publishing. Review authors should generally search sources such as dissertations and conference abstracts (see MECIR Box 4.3.e).

Review authors may also consider searching the internet, handsearching journals and searching full texts of journals electronically where available (see online Technical Supplement for details). They should examine previous reviews on the same topic and check reference lists of included studies and relevant systematic reviews (see MECIR Box 4.3.e).

MECIR Box 4.3.e Relevant expectations for conduct of intervention reviews

4.4 Designing search strategies

4.4.1 Introduction to search strategies

This section highlights some of the issues to consider when designing search strategies. Designing search strategies can be complex and the section does not fully address the many complexities in this area. Review teams will benefit from the skills and expertise of a medical/healthcare librarian or information specialist. Many of the issues highlighted relate to both the subject aspects of the search (e.g. the PICO elements) and to the study design (e.g. randomized trials). For a search to be robust, both aspects require attention to be sure that relevant records are not missed.

Issues to consider in planning a search include:

• the nature or type of the intervention(s) being assessed;
• the complexity of the review question and the need to consider additional conceptual frameworks (see Chapter 3 and Chapter 17);
• the time period when any evaluations of the interventions may have taken place (as specified in the review protocol) (see Section 4.4.5);
• any geographic considerations, such as the need to search the African Index Medicus for studies relating to African populations or the Chinese literature for studies in Chinese herbal medicine (see online Technical Supplement);
• whether the review is limited to randomized trials or other study designs are eligible (see Chapter 24);
• whether a validated methodological search filter (for specific study designs) is available (see Section 4.4.7);
• whether unpublished data are to be sought specifically, see Sections 4.3.2, 4.3.3 and 4.3.4; and
• whether the review has specific eligibility criteria around study design to address adverse effects (see Chapter 19), economic evidence (see Chapter 20) or qualitative evidence (see Chapter 21), in which case searches to address these criteria should be undertaken (see MECIR Box 4.4.a).

Further evidence-based information about designing search strategies can be found on the SuRe Info portal, which is updated twice per year.

MECIR Box 4.4.a Relevant expectations for conduct of intervention reviews

4.4.2 Structure of a search strategy

The starting point for developing a search strategy is to consider the main concepts being examined in a review. This is often referred to as PICO – that is Patient (or Participant or Population or Problem), Intervention, Comparison and Outcomes (Richardson et al 1995): see also Chapter 2 and Chapter 3 for guidance on developing and refining PICO definitions that will be operationalized in the search strategy. Examples are provided in the appendices to the Cochrane Information Specialists’ Handbook (Cochrane Information Specialist Support Team 2021c). For a Cochrane Review, the review objective should provide the PICO concepts, and the eligibility criteria for studies to be included will further assist in the selection of appropriate subject headings and text words for the search strategy.

The structure of search strategies in bibliographic databases should be informed by the main concepts of the review (see Chapter 3), using appropriate elements from PICO and study design (see MECIR Box 4.4.b). It is usually unnecessary, however, and may even be undesirable, to search on every aspect of the review’s clinical question (Frandsen et al 2020). Although a research question may specify particular comparators or outcomes, these concepts may not be well described in the title or abstract of an article and are often not well indexed with controlled vocabulary terms. Therefore, in general databases, such as MEDLINE, a search strategy will typically have three sets of terms: (i) terms to search for the health condition of interest, i.e. the population; (ii) terms to search for the intervention(s) evaluated; and (iii) terms to search for the types of study design to be included. Typically, a broad set of search terms will be gathered for each concept and combined with the OR Boolean operator to achieve sensitivity within concepts. The results for each concept are then combined using the AND Boolean operator, to ensure each concept is represented in the final search results.

It is important to consider the structure of the search strategy on a question-by-question basis. In some cases it is possible and reasonable to search for the comparator, for example if the comparator is explicitly placebo; in other cases the outcomes may be particularly well defined and consistently reported in abstracts. The advice on whether or not to search for outcomes for adverse effects differs from the advice given above (see Chapter 19).

MECIR Box 4.4.b Relevant expectations for conduct of intervention reviews

Some search strategies may not easily divide into the structure suggested, particularly for reviews addressing complex or unknown interventions, or diagnostic tests (Huang et al 2006, Irvin and Hayden 2006, Petticrew and Roberts 2006, de Vet et al 2008, Booth 2016) or using specific approaches such as realist reviews which may require iterative searches and multiple search strategies (Booth et al 2020). Cochrane Reviews of public health interventions and of qualitative data may adopt very different search approaches to those described here (Lorenc et al 2014, Booth 2016) (see Chapter 17 on intervention complexity, and Chapter 21 on qualitative evidence). Some options to explore for such situations include:

• use a single concept such as searching for the intervention alone (European Food Safety Authority 2010);
• break a concept into two or more subconcepts;
• use a multi-stranded or multi-faceted approach that uses a series of searches, with different combinations of concepts, to capture a complex research question (Lefebvre et al 2013);
• use a variety of different search approaches to compensate for when a specific concept is difficult to define (Shemilt et al 2014);
• use iterative searches (Bravata et al 2005, Zwakman et al 2018, Booth et al 2020); or
• use citation searching on key articles in addition to a database search (Haddaway et al 2015, Hinde and Spackman 2015) (see online Technical Supplement).

4.4.3 Sensitivity versus precision

Searches for systematic reviews aim to be as extensive as possible in order to ensure that as many of the relevant studies as possible are included in the review. It is, however, necessary to strike a balance between striving for comprehensiveness and maintaining relevance when developing a search strategy.

The properties of searches are often quantified using ‘sensitivity’ (also called ‘recall’) and ‘precision’ (see Table 4.4.a). Sensitivity is defined as the number of relevant reports identified divided by the total number of relevant reports in the resource. Precision is defined as the number of relevant reports identified divided by the total number of reports identified. Increasing the comprehensiveness (or sensitivity) of a search will reduce its precision and will usually retrieve more non-relevant reports.

Searches for Cochrane Reviews should seek to maximize sensitivity whilst striving for reasonable precision (see MECIR Box 4.4.b). Article abstracts identified through a database search can usually be screened very quickly to ascertain potential relevance. At a conservatively estimated reading rate of one or two abstracts per minute, the results of a database search can be screened at the rate of 60–120 per hour (or approximately 500–1000 over an 8-hour period), so the high yield and low precision associated with systematic review searching may not be as daunting as it might at first appear in comparison with the total time to be invested in the review.

Table 4.4.a Sensitivity and precision of a search

4.4.4 Controlled vocabulary and text words

This section should be read in conjunction with Section 3.2 and its subsections in the online Technical Supplement. MEDLINE and Embase (and many other databases) can be searched using a combination of two retrieval approaches. One is based on text words, that is terms occurring in the title, abstract or other relevant fields available in the database. The other is based on standardized subject terms assigned to the references either by indexers (specialists who appraise the articles and describe their topics by assigning terms from a specific thesaurus or controlled vocabulary) or automatically using automated indexing approaches. Searches for Cochrane Reviews should use an appropriate combination of these two approaches, i.e. text words and controlled vocabulary (see MECIR Box 4.4.c). Approaches for identifying text words and controlled vocabulary to combine appropriately within a search strategy, including text mining approaches, are presented in the online Technical Supplement.

MECIR Box 4.4.c Relevant expectations for conduct of intervention reviews

4.4.5 Language, date and document format restrictions

Searches should capture as many studies as possible that meet the eligibility criteria, ensuring that relevant time periods and sources are covered and not restricted by language or publication status (see MECIR Box 4.3.a). Review authors should justify the use of any restrictions in the search strategy on publication date and publication format (see MECIR Box 4.4.d).

To reduce the risk of introducing bias, searches should not be restricted by language. Recommendations for rapid reviews searches to limit publication language to English and add other languages only when justified (Garritty et al 2021) are supported by evidence that excluding non-English studies does not change the conclusions of most systematic reviews (Morrison et al 2012, Jiao et al 2013, Hartling et al 2017, Nussbaumer-Streit et al 2020). However, exceptions that non-English studies do influence review findings have been observed for complementary and alternative medicine (Moher et al 2003, Pham et al 2005, Wu et al 2013), psychiatry, rheumatology and orthopaedics (Egger et al 2003).

Studies have identified a risk of introducing bias by including lower quality, non-English language trials in systematic reviews (Jüni et al 2002, Egger et al 2003), but similar evaluations found only minor quality differences between reports of English and non-English language trials (Moher et al 2003). Additionally, when searches are limited to English or to databases containing only English-language articles, there is a risk that eligible studies may be missed from countries where a particular intervention of interest is more common (e.g. traditional Chinese medicines) (Pilkington et al 2005, Morrison et al 2012). For further discussion of these issues see Chapter 13.

Particularly when resources and time are available, the inclusion of non-English studies in systematic reviews is recommended to minimize the risk of language bias (Egger et al 1997, Pilkington et al 2005, Morrison et al 2012). Consequently, Cochrane author teams should plan at the protocol stage not to restrict the search by language (see MECIR Box 4.3.a). It has also been argued that, when language restrictions are justified, these should not be imposed by limiting the search but by including language as an eligibility criterion during study selection (Pieper and Puljak 2021).

If a Cochrane Review team requires help with translation of or data extraction from non-English language reports of studies, they should seek assistance to do so (this is a common task for which volunteer assistance can be sought via Cochrane Engage platform, accessible to both Cochrane and non-Cochrane review teams). Where it is not possible to extract the relevant information and data from non-English language reports, readers should be informed of the existence of other possibly relevant reports by adding such reports to ‘studies awaiting classification’ rather than ‘excluded studies’. This information should be reflected in the PRISMA flow diagram (or, if there is no flow diagram, then in the text of the review) as ‘studies awaiting classification’.

Date limits may be used to focus searches (Cooper et al 2018a) as long as the restriction is reported and justified (Egger et al 2003) (see MECIR Box 4.4.d). Further use of a supportive narrative may help explain why a particular date restriction was applied (Craven and Levay 2011, Cooper et al 2018b). For example, a database date restriction of 1989-current for a review of nurse-led community training of epinephrine autoinjectors is justified because this is the approval date of the first device (Center for Drug Evaluation and Research 1989). A date limit may be safely applied in this case as any references published before this date would not meet the review’s selection criteria. Conversely, arbitrary date restrictions intended to reduce search yield (e.g. last ten years) should be avoided as a potential source of bias.

Caution should be exercised when designing database search strategies with date restrictions. Information specialists should be aware of the various date fields available from database providers (e.g. create date, entry date, last update date, publication date) as well as the coverage dates of the datafiles searched. It may be necessary to search additional sources or datafiles to ensure adequate coverage of the date period of interest for the review. To account for inconsistent publication dates in database records (e.g. a record for an electronic version of a publication may have an earlier publication date than the print version), search strategies should be restricted to a wider date range than the period of interest for the review.

As any information about an eligible study may contain valuable details for analysis, document format restrictions should not be applied to systematic review searches. For example, excluding letters is not recommended because letters may contain important additional information relating to an earlier trial report or new information about a trial not reported elsewhere (Iansavichene et al 2008). In addition, articles indexed as ‘Comments’ should not be routinely excluded without further examination as these may contain early warnings of possible future retraction (see Section 4.4.6).

As with comments and letters, preprints (versions of scientific articles that precede formal peer review and publication in a journal) should also be considered a potentially relevant source of study evidence. Recent and widespread availability of preprints has resulted from an urgent demand for emerging evidence during the COVID-19 pandemic (Gianola et al 2020, Kirkham et al 2020, Callaway 2021, Fraser et al 2021). As study data are often reported in multiple publications and may be reported differently in each (Oikonomidi et al 2020), efforts to identify all reports for eligible studies, regardless of publication format, are necessary to support subsequent stages of the review process to select, assess and analyse complete study data.

MECIR Box 4.4.d Relevant expectations for conduct of intervention reviews

4.4.6 Identifying fraudulent studies, other retracted publications, errata and comments

When considering the eligibility of studies for inclusion in a Cochrane Review, it is important to be aware that some studies may have been found to contain errors or to be fraudulent or may, for other reasons, have been corrected or retracted since publication. Review authors should examine any relevant retraction statements and errata for information (MECIR Box 4.4.e). This applies both to ‘new’ studies identified for inclusion in a review and to studies that are already included in a review when the review is updated. For review updates, it is important to search MEDLINE and Embase for the latest version of the citations to the records for the (previously) included studies, in case they have since been corrected or retracted.

Errata are published to correct unintended errors (accepted as errors by the author(s)) that do not invalidate the conclusions of the article. Retractions are defined by the Committee on Publication Ethics (COPE) Council’s retraction guidelines (Committee on Publication Ethics (COPE) Council 2019) as “… a mechanism for correcting the literature and alerting readers to articles that contain such seriously flawed or erroneous content or data that their findings and conclusions cannot be relied upon. Unreliable content or data may result from honest error, naïve mistakes, or research misconduct.” Comments are published under a range of circumstances including when errors are suggested by others and also for early concerns regarding suspected misconduct.

Including data from studies that are fraudulent or studies that include errors can have an impact on the overall estimates in systematic reviews. There is an increasing awareness of the importance of not including retracted studies or those with significant errata in systematic reviews and how best to avoid this (Royle and Waugh 2004, Wright and McDaid 2011, Decullier et al 2014). A recent study, however, showed that even when review authors suspect research misconduct, including data falsification, in the trials that they are considering including in their systematic reviews, they do not always report it (Elia et al 2016). Details of how to identify fraudulent studies, other retracted publications, errata and comments are described in the online Technical Supplement.

MECIR Box 4.4.e Relevant expectations for conduct of intervention reviews

4.4.7 Search filters

Search filters are search strategies that are designed to retrieve specific types of records, such as those of a particular methodological design. When searching for randomized trials in humans, a validated filter should be used to identify studies with the appropriate design (see MECIR Box 4.4.f). Filters to identify randomized trials for CENTRAL have been developed specifically for databases such as MEDLINE, Embase and CINAHL Plus: see the online Technical Supplement for details. CENTRAL, however, aims to contain only reports with study designs possibly relevant for inclusion in Cochrane Reviews, so searches of CENTRAL should not use a trials ‘filter’ or be limited to human studies.

The InterTASC Information Specialists’ Sub-Group Search Filter Resource offers a collection of search filters, focusing predominantly on methodological search filters and providing critical appraisals of some of these filters. The site includes, amongst others, filters for identifying systematic reviews, randomized and non-randomized studies and qualitative research in a range of databases and across a range of service providers (Glanville et al 2019). For further discussion around the design and use of search filters, see the online Technical Supplement.

MECIR Box 4.4.f Relevant expectations for conduct of intervention reviews

4.4.8 Peer review of search strategies

It is strongly recommended that search strategies should be peer reviewed before the searches are run. Peer review of search strategies is increasingly recognized as a necessary step in designing and executing high-quality search strategies to identify studies for possible inclusion in systematic reviews. As discussed elsewhere (Lefebvre and Duffy 2021), the following organizations and documents advocate peer review of searches: The Agency for Healthcare Research and Quality (AHRQ) in the US, the Centre for Reviews and Dissemination in the UK, the European Network for Health Technology Assessment (EUnetHTA), the Institute for Quality and Efficiency in Health Care (IQWiG) in Germany, the Institute of Medicine in the US, the National Institute of Health and Care Excellence (NICE) in the UK, the Preferred Reporting Items for Systematic reviews and Meta-Analyses – Extension for Searches (PRISMA-S Extension) and the PRISMA 2020 statement and explanation and elaboration documents (Centre for Reviews and Dissemination 2009, Institute of Medicine 2011, Agency for Healthcare Research and Quality 2014, EUnetHTA JA3WP6B2-2 Authoring Team 2019, Institute for Quality and Efficiency in Health Care 2020, National Institute for Health and Care Excellence 2020, Page et al 2021b, Page et al 2021a, Rethlefsen et al 2021).

Studies have shown that errors occur in the search strategies underpinning systematic reviews and that search strategies are not always conducted or reported to a high standard (Mullins et al 2014, Layton 2017, Salvador-Olivan et al 2019). This has also been shown to be the case within some Cochrane Reviews (Franco et al 2018). The PRISMA-S Extension states that authors “should strongly consider having the search strategy peer reviewed by an experienced searcher, informational specialist, or librarian” and encourages authors to consider using the Peer Review of Electronic Search Strategies (PRESS) Guideline Statement (Rethlefsen et al 2021). Research has shown that peer review using a specially designed checklist can improve the quality of searches both in systematic reviews (Relevo and Paynter 2012, Spry et al 2013) and in rapid reviews (Spry et al 2013, Spry and Mierzwinski-Urban 2018). An evidence-based checklist such as the PRESS Evidence-Based Checklist should be used to assess which elements are important in peer review of electronic search strategies (McGowan et al 2016a, McGowan et al 2016b). The PRESS checklist covers not only the technical accuracy of the strategy (line numbers, spellings, etc.), but also whether the search strategy addresses all relevant aspects of the protocol and has interpreted the research question appropriately.

It is recommended that authors provide information on the search strategy development and peer review processes. The PRISMA 2020 explanation and elaboration article and the PRISMA-S Extension provide guidance on how and where authors should describe the processes used to develop and validate or peer review the search strategy (Page et al 2018, Rethlefsen et al 2021). For example, the PRISMA 2020 explanation and elaboration article states that “the description of the search strategy development process might include details of the approaches used to identify keywords, synonyms, or subject indexing terms used in the search strategies or any processes used to validate or peer review the search strategies”, and “if the search strategy was peer reviewed, report the peer review process used and specify any tool used, such as the Peer Review of Electronic Search Strategies (PRESS) checklist” (Page et al 2021b). In the example for Item 7 (Search strategy) of the PRISMA 2020 checklist, the authors propose using the following statement: “The strategy was developed by an information specialist and the final strategies were peer reviewed by an experienced information specialist within our team” (Page et al 2021a). The PRISMA-S Extension advocates that the use of peer review be reported and described in the methods section, and proposes the following statement in Item 14 (Peer review: Describe any search peer review process): “The strategies were peer reviewed by another senior information specialist prior to execution using the PRESS Checklist” (Rethlefsen et al 2021). For Cochrane Reviews, the names, credentials, and institutions of the peer reviewers of the search strategies should be noted in the review (with their permission) in the Acknowledgments section.

4.4.9 Alerts

Alerts, also called literature surveillance services, ‘push’ services or SDIs (selective dissemination of information), are an excellent method of staying up to date with the medical literature currently being published, as a supplement to designing and running specific searches for specific reviews. In practice, alerts are based on a previously developed search strategy, which is saved in a personal account on the database platform (e.g. ‘My EBSCOhost – search alerts’ on EBSCO, ‘My searches & alerts’ on Ovid and ‘MyNCBI – saved searches’ on PubMed). These saved strategies filter the content as the database is being updated with new information. The account owner is notified (usually via email) when new publications meeting their specified search parameters are added to the database. In the case of PubMed, the alert can be set up to be delivered weekly or monthly, or in real-time and can comprise email or RSS feeds.

For review authors, alerts are a useful tool to help monitor what is being published in their review topic after the original search has been conducted. By following the alert, authors can become aware of a new study that meets the review’s eligibility criteria, and decide either to include it in the review immediately or mention it as a reference pending classification for inclusion during the next review update (see online Chapter IV). Authors should consider setting up alerts so that the review can be as current as possible at the time of publication.

Another way of attempting to stay current with the literature as it emerges is by using alerts based on journal tables of contents (TOCs). These usually cannot be specifically tailored to the information needs in the same way as search strategies developed to cover a specific topic. They can, however, be a good way of trying to keep up to date on a more general level by monitoring what is currently being published in journals of interest. Many journals, even those that are available by subscription only, offer TOC alert services free of charge. In addition, a number of publishers and organizations offer TOC services (see online Technical Supplement). Use of TOCs is not proposed as a single alternative to the various other methods of study identification necessary for undertaking systematic reviews, rather as a supplementary method. (See also Chapter 22, Section 22.2 for a discussion of new technologies to support evidence surveillance in the context of ‘living’ systematic reviews.)

Alerts should also be considered for sources beyond databases and journal TOCs, such as trials register resources and regulatory information.

4.4.10 Timing of searches

The published review should be as up to date as possible. Searches for all the relevant databases should be rerun prior to publication, if the initial search date is more than 12 months (preferably six months) from the intended publication date (see MECIR Box 4.4.g). This is also good practice for searches of non-database sources. The results should also be screened to identify potentially eligible studies. Ideally, the studies should be incorporated fully in the review. If not, then the potentially eligible studies will need to be reported as references under ‘studies awaiting classification’ (or under ‘ongoing studies’ if they are not yet completed).

MECIR Box 4.4.g Relevant expectations for conduct of intervention reviews

4.4.11 When to stop searching

Developing a search is often an iterative and exploratory process. It involves exploring trade-offs between search terms and assessing their overall impact on the sensitivity and precision of the search. It is often difficult to decide in a scientific or objective way when a search is complete and search strategy development can stop. The ability to decide when to stop typically develops through experience of developing many strategies. Suggestions for stopping rules have been made around the retrieval of new records, for example to stop if adding in a series of new terms to a database search strategy yields no new relevant records, or if precision falls below a particular cut-off point (Chilcott et al 2003). Stopping might also be appropriate when the removal of terms or concepts results in missing relevant records. Another consideration is the amount of evidence that has already accrued: in topics where evidence is scarce, authors might need to be more cautious about deciding when to stop searching. Although many methods have been described to assist with deciding when to stop developing the search, there has been little formal evaluation of the approaches (Booth 2010, Arber and Wood 2021).

At a basic level, investigation is needed as to whether a strategy is performing adequately. One simple test is to check whether the search is finding the publications that have been recommended as key publications or that have been included in other similar reviews (EUnetHTA JA3WP6B2-2 Authoring Team 2019). It is not enough, however, for the strategy to find only those records, otherwise this might be a sign that the strategy is biased towards known studies and other relevant records might be being missed. In addition, citation searches (see online Technical Supplement Section 1.1.4) and reference checking (see online Technical Supplement Section 1.3.4) are useful checks of strategy performance. If those additional methods are finding documents that the searches have already retrieved, but that the team did not necessarily know about in advance, then this is one sign that the strategy might be performing adequately. Also, an evidence-based checklist such as the PRESS Evidence-Based Checklist (McGowan et al 2016b) should be used to assess whether the search strategy is adequate (see Section 4.4.8). If some of the PRESS dimensions seem to be missing without adequate explanation or arouse concerns, then the search may not yet be complete.

Statistical techniques can be used to assess performance, such as capture-recapture (Spoor et al 1996, Ferrante di Ruffano et al 2012) (also known as capture-mark-recapture; (Kastner et al 2009, Lane et al 2013), or the relative recall technique (Sampson et al 2006, Sampson and McGowan 2011). Kastner suggests the capture-mark-recapture technique merits further investigation since it could be used to estimate the number of studies in a literature prospectively and to determine where to stop searches once suitable cut-off levels have been identified. Kastner’s approach involves searching databases, conducting record selection, calculating capture-mark-recapture and then making decisions about whether further searches are necessary. This would entail potentially an iterative search and selection process. Capture-recapture needs results from at least two searches to estimate the number of missed studies. Further investigation of published prospective techniques seems warranted to learn more about the potential benefits.

Relative recall (Sampson et al 2006, Sampson and McGowan 2011) requires a range of searches to have been conducted so that the relevant studies have been built up by a set of sensitive searches. The performance of the individual searches can then be assessed in each individual database by determining how many of the studies that were deemed eligible for the evidence synthesis and were indexed within a database, can be found by the database search used to populate the synthesis. If a search in a database did not perform well and missed many studies, then that search strategy is likely to have been suboptimal. If the search strategy found most of the studies that were available to be found in the database, then it was likely to have been a sensitive strategy. Assessments of precision could also be made, but these mostly inform future search approaches since they cannot affect the searches and record assessment already undertaken. Relative recall may be most useful at the end of the search process since it relies on the achievement of several searches to make judgements about the overall performance of strategies.

In evidence synthesis involving qualitative data, searching is often more organic and intertwined with the analysis such that the searching stops when new information ceases to be identified (Booth 2016). The reasons for stopping need to be documented and it is suggested that explanations or justifications for stopping may centre around saturation (Booth 2016). Further information on searches for qualitative evidence can be found in Chapter 21.

4.5 Documenting and reporting the search process

Review authors should document the search process in enough detail to ensure that it can be reported correctly in the review (see MECIR Box 4.5.a). The searches of all the databases should be reproducible to the extent that this is possible. By documenting the search process, we refer to internal record-keeping, which is distinct from reporting the search process in the review (discussed in online Chapter III).

MECIR Box 4.5.a Relevant expectations for conduct of intervention reviews

Medical/healthcare librarians and information specialists involved with the review should draft, or at least comment on, the search strategy sections of the review prior to publication.

Suboptimal reporting of systematic review search activities and methods has been observed (Sampson et al 2008, Roundtree et al 2009, Niederstadt and Droste 2010). Research has also shown a lack of compliance with guidance in the Handbook with respect to search strategy description in published Cochrane Reviews (Sampson and McGowan 2006, Yoshii et al 2009, Franco et al 2018). The lack of consensus regarding optimal reporting has been a challenge with respect to the values of transparency and reproducibility. The PRISMA-Search (PRISMA-S) Extension (Rethlefsen et al 2021), an extension to the PRISMA Statement (Page et al 2021a, Page et al 2021b), addresses the reporting of search strategies in systematic reviews. PRISMA-S (together with the major revision of PRISMA itself) provides enough detail and specific examples for systematic review authors to report search methods and information sources in a clear, reproducible way. In Box 2 of the PRISMA 2020 guidance under “Noteworthy changes to the PRISMA 2009 statement” the guidance has been strengthened to stipulate: “Modification of the ‘Search’ item to recommend authors present full search strategies for all databases, registers and websites searched, not just at least one database (see item #7)”. This brings the PRISMA 2020 guidance more into line with Cochrane standards for reporting of database search strategies.

There is also a recommendation in the PRISMA 2020 guidance (see item 27) that “authors state whether data used in the review are publicly available and if so, where they can be accessed”(Page et al 2021a). These recommendations may influence record keeping practices of searchers.

It is recommended that review authors seek guidance from their medical/healthcare librarian or information specialist at the earliest opportunity with respect to documenting the search process (Rethlefsen et al 2015, Meert et al 2016). For Cochrane Reviews, the bibliographic database search strategies should be copied and pasted into an appendix exactly as run and in full, together with the search set numbers and the total number of records retrieved by each search strategy. The search strategies should not be re-typed, because this can introduce errors. The same process is also good practice for searches of trials registers and other sources, where the interface used, such as introductory or advanced, should also be specified. Creating a report of the search process can be accomplished through methodical documentation of the steps taken by the searcher. This need not be onerous if suitable record keeping is performed during the process of the search, but it can be nearly impossible to recreate post hoc. Many database interfaces have facilities for search strategies to be saved online or to be emailed; an offline copy in text format should also be saved. For some databases, taking and saving a screenshot of the search may be the most practical approach (Rader et al 2014).

Documenting the searching of sources other than databases, including the search terms used, is also required if searches are to be reproducible (Atkinson et al 2015, Chow 2015, Witkowski and Aldhouse 2015).

Details about contacting experts or manufacturers, searching reference lists, scanning websites, and decisions about search iterations can be produced as an appendix in the final document and used for future updates. The purpose of search documentation is transparency, internal assessment, and reference for any future update. It is important to plan how to record searching of sources other than databases since some activities (contacting experts, reference list searching, and forward citation searching) will occur later on in the review process after the database results have been screened (Rader et al 2014). The searcher should record any correspondence on key decisions and report a summary of this correspondence alongside the search strategy in a search narrative. The narrative describes the major decisions that shaped the strategy and can give a peer reviewer an insight into the rationale for the search approach (Craven and Levay 2011). A worked example of a search narrative is available (Cooper et al 2018b).

It is particularly important to save locally or file print copies of any information found on the Internet, such as information about ongoing and/or unpublished trials, as this information may no longer be accessible at the time the review is written. Local copies should be stored in a structured way to allow retrieval when needed. There are also web-based tools which archive webpage content for future reference, such as WebCite (Eysenbach and Trudel 2005). The results of web searches will not be reproducible to the same extent as bibliographic database searches because web content and search engine algorithms frequently change, and search results can differ between users due to a general move towards localization and personalization (Cooper et al 2021b). It is still important, however, to document the search process to ensure that the methods used can be transparently reported (Briscoe 2018). In cases where a search engine retrieves more results than it is practical to screen in full (it is rarely practical to search thousands of web results, as the precision of web searches is likely to be relatively low), the number of results that are documented and reported should be the number that were screened rather than the total number (Dellavalle et al 2003, Bramer 2016).

Decisions should be documented for all records identified by the search. Details of the flow of studies from the number(s) of references identified in the search to the number of studies included in the review will need to be reported in the final review, ideally using a flow diagram such as that proposed in the PRISMA guidance (see online Chapter III); these can be generated using software including Covidence, DistillerSR, EPPI-Reviewer, the METAGEAR package for R, the PRISMA Flow Diagram Generator, and RevMan. A table of ‘Characteristics of excluded studies’ will also need to be presented (see Section 4.6.5). Numbers of records are sufficient for exclusions based on initial screening of titles and abstracts. Broad categorizations are sufficient for records classed as potentially eligible during an initial screen of the full text. Authors will need to decide for each review when to map records to studies (if multiple records refer to one study). The flow diagram records initially the total number of records retrieved from various sources, then the total number of studies to which these records relate. Review authors need to match the various records to the various studies in order to complete the flow diagram correctly. Lists of included and excluded studies must be based on studies rather than records (see also Section 4.6.1).

4.6 Selecting studies

4.6.1 Studies (not reports) as the unit of interest

A Cochrane Review is a review of studies that meet pre-specified eligibility criteria. Since each study may have been reported in several articles, abstracts or other reports, an extensive search for studies for the review may identify many reports for each potentially relevant study. Two distinct processes are therefore required to determine which studies can be included in the review. One is to link together multiple reports of the same study; and the other is to use the information available in the various reports to determine which studies are eligible for inclusion. Although sometimes there is a single report for each study, it should never be assumed that this is the case.

As well as the studies that inform the systematic review, other studies will also be identified and these should be recorded or tagged as they are encountered, so that they can be listed in the relevant tables in the review:

• records of ongoing trials for which results (either published or unpublished) are not (yet) available; and
• records of studies which seem to be eligible but for which data are incomplete or the publication related to the record could not be obtained.

4.6.2 Identifying multiple reports from the same study

Duplicate publication can introduce substantial biases if studies are inadvertently included more than once in a meta-analysis (Tramèr et al 1997). Duplicate publication can take various forms, ranging from identical manuscripts to reports describing different outcomes of the study or results at different time points (von Elm et al 2004). The number of participants may differ in the different publications. It can be difficult to detect duplicate publication and some ‘detective work’ by the review authors may be required.

Some of the most useful criteria for comparing reports are:

• trial identification numbers (e.g. ClinicalTrials.gov Identifier (NCT number); ISRCTN; Universal Trial Number (UTN) (assigned by the ICTRP); other identifiers such as those from the sponsor);
• author names (most duplicate reports have one or more authors in common, although this is not always the case);
• location and setting (particularly if institutions, such as hospitals, are named);
• specific details of the interventions (e.g. dose, frequency);
• numbers of participants and baseline data; and
• date and duration of the study (which can also clarify whether different sample sizes are due to different periods of recruitment).

Where uncertainties remain after considering these and other factors, it may be necessary to correspond with the authors of the reports.

Multiple reports of the same study should be collated, so that each study, rather than each report, is the unit of interest in the review (see MECIR Box 4.6.a). Review authors will need to choose and justify which report (the primary report) to use as a source for study results, particularly if two reports include conflicting results. They should not discard other (secondary) reports, since they may contain additional outcome measures and valuable information about the design and conduct of the study.

MECIR Box 4.6.a Relevant expectations for conduct of intervention reviews

4.6.3 A typical process for selecting studies

A typical process for selecting studies for inclusion in a review is as follows (the process should be detailed in the protocol for the review):

1. Merge search results from different sources using reference management software, and remove duplicate records of the same report (i.e. records reporting the same journal title, volume and pages).
2. Examine titles and abstracts to remove obviously irrelevant reports (authors should generally be over-inclusive at this stage).
3. Retrieve the full text of the potentially relevant reports.
4. Link together multiple reports of the same study (see Section 4.6.2).
5. Examine full-text reports for compliance of studies with eligibility criteria.
6. Correspond with investigators, where appropriate, to clarify study eligibility (it may be appropriate to request further information, such as missing methods information or results, at the same time). If studies remain incomplete/unobtainable they should be tagged/recorded as incomplete, and should be listed in the table of ‘Characteristics of studies awaiting classification’ in the review.
7. Make final decisions on study inclusion and proceed to data collection.
8. Tag or record any ongoing trials which have not yet been reported so that they can be added to the ongoing studies table.

Note that studies should not be omitted from a review solely on the basis of measured outcome data not being reported (see MECIR Box 4.6.b and Chapter 13).

MECIR Box 4.6.b Relevant expectations for conduct of intervention reviews

4.6.4 Implementation of the selection process

Decisions about which studies to include in a review are among the most influential decisions that are made in the review process and they involve judgement. Use (at least) two people working independently to determine whether each study meets the eligibility criteria. Ideally, screening of titles and abstracts to remove irrelevant reports should also be done in duplicate by two people working independently (although it is acceptable that this initial screening of titles and abstracts is undertaken by only one person). It is essential, however, that two people working independently are used to make a final determination as to whether each study considered possibly eligible after title/abstract screening meets the eligibility criteria based on the full text of the study report(s) (see MECIR Box 4.6.c).

MECIR Box 4.6.c Relevant expectations for conduct of intervention reviews

It has been shown that using at least two authors may reduce the possibility that relevant reports will be discarded (Edwards et al 2002, Waffenschmidt et al 2019, Gartlehner et al 2020) although other case reports have suggested single screening approaches may be adequate (Doust et al 2005, Shemilt et al 2016). Opportunities for screening efficiencies seem likely to become available through promising developments in single human screening in combination with machine learning approaches (O'Mara-Eves et al 2015).

Experts in a particular area frequently have pre-formed opinions that can bias their assessment of both the relevance and validity of articles (Cooper and Ribble 1989, Oxman and Guyatt 1993). Thus, while it is important that at least one author is knowledgeable in the area under review, it may be an advantage to have a second author who is not a content expert.

Disagreements about whether a study should be included can generally be resolved by discussion. Often the cause of disagreement is a simple oversight on the part of one of the review authors. When the disagreement is due to a difference in interpretation, this may require arbitration by another person. Occasionally, it will not be possible to resolve disagreements about whether to include a study without additional information. In these cases, authors may choose to categorize the study in their review as one that is awaiting assessment until the additional information is obtained from the study authors.

A single failed eligibility criterion is sufficient for a study to be excluded from a review. In practice, therefore, eligibility criteria for each study should be assessed in order of importance, so that the first ‘no’ response can be used as the primary reason for exclusion of the study, and the remaining criteria need not be assessed. The eligibility criteria order may be different in different reviews and they do not always need to be the same.

For most reviews it will be worthwhile to pilot test the eligibility criteria on a sample of reports (say six to eight articles, including ones that are thought to be definitely eligible, definitely not eligible and doubtful). The pilot test can be used to refine and clarify the eligibility criteria, train the people who will be applying them and ensure that the criteria can be applied consistently by more than one person.

For Cochrane Reviews the selection process must be documented in sufficient detail to be able to complete a flow diagram and a table of ‘Characteristics of excluded studies’ (see MECIR Box 4.6.d). During the selection process it is crucial to keep track of the number of references and subsequently the number of studies so that a flow diagram can be constructed. The decision and reasons for exclusion can be tracked using reference management software, a simple document or spreadsheet, or using specialist systematic review software (see Section 4.6.6.1).

MECIR Box 4.6.d Relevant expectations for conduct of intervention reviews

4.6.5 Selecting ‘excluded studies’

A Cochrane Review includes a list of excluded studies called ‘Characteristics of excluded studies’, detailing the specific reason for exclusion for any studies that a reader might plausibly expect to see among the included studies. This covers all studies that may, on the surface, appear to meet the eligibility criteria but which, on further inspection, do not. It also covers those that do not meet all of the criteria but are well known and likely to be thought relevant by some readers. By listing such studies as excluded and giving the primary reason for exclusion, the review authors can show that consideration has been given to these studies. The list of excluded studies should be as brief as possible. It should not list all of the reports that were identified by an extensive search. It should not list studies that obviously do not fulfil the eligibility criteria for the review, such as ‘Types of studies’, ‘Types of participants’, and ‘Types of interventions’. In particular, it should not list studies that are obviously not randomized if the review includes only randomized trials. Based on a sample, undertaken in 2017/2018 by one of the authors (JT), of approximately 60% of the intervention reviews in the Cochrane Library which included randomized trials only, the average number of studies listed in the ‘excluded studies’ table was 30.

4.6.6 Software support for selecting studies

An extensive search for eligible studies in a systematic review can often identify thousands of records that need to be manually screened. Selecting studies from within these records can be a particularly time-consuming, laborious and logistically challenging aspect of conducting a systematic review. These and other challenges have led to the development of various software tools (and approaches for using ‘generic’ tools) that offer support for the selection process.

Broadly, software to support selecting studies can be classified as:

• systems that support the study selection process, typically involving multiple reviewers (see Section 4.6.6.1); and
• tools and techniques based on text mining and/or machine learning, which aim to semi- or fully-automate the selection process (see Section 4.6.6.2).

Software to support the selection process, along with other stages of a systematic review, including text mining tools, can be identified using the Systematic Review Toolbox. The SR Toolbox is a community driven, web-based catalogue of tools that provide support for systematic reviews (Marshall and Brereton 2015).

4.6.6.1 Software for managing the selection process

Managing the selection process can be challenging, particularly in a large-scale systematic review that involves multiple reviewers. Basic productivity tools can help (such as word processors, spreadsheets, and reference management software), and several purpose-built systems that support multiple concurrent users are also available that offer support for the study selection process. Software for managing the selection process can be identified using the Systematic Review Toolbox mentioned above.

Compatibility with other software tools used in the review process (such as RevMan) may be a consideration when selecting a tool to support study selection. Covidence and EPPI-Reviewer are Cochrane-preferred tools, and are likely to have the strongest integration with RevMan. Should specialist software not be available, Bramer and colleagues have developed a method for using the widely available software EndNote X7 for managing the screening process (Bramer et al 2017).

4.6.6.2 Automating the selection process

Research into automating the study selection process through machine learning and text mining has received considerable attention over recent years, resulting in the development of various tools and techniques for reviewers to consider. The use of automated tools has the potential to reduce the workload involved with selecting studies significantly (Thomas et al 2017). For example, research suggests that adopting automation can reduce the need for manual screening by at least 30% and possibly more than 90%, although sometimes at the cost of up to a 5% reduction in sensitivity (O'Mara-Eves et al 2015).

Machine learning models (or ‘classifiers’) can be built where sufficient data are available. Of particular practical use to Cochrane Review authors is a classifier (the ‘RCT Classifier’) that can identify reports of randomized trials based on titles and abstracts. The classifier is highly accurate because it is built on a large dataset of hundreds of thousands of records screened by Cochrane Crowd, Cochrane’s citizen science platform, where contributors help to identify and describe health research (Marshall et al 2018, Noel-Storr et al 2021, Thomas et al 2021). Guidance on using the RCT Classifier in Cochrane Reviews, for example to exclude studies already flagged as not being randomized trials, or to access Cochrane Crowd to assist with screening, is available from the Cochrane Information Specialists’ Handbook (Cochrane Information Specialist Support Team 2021d).

Cochrane has also implemented a screening workflow called Screen4Me. This workflow incorporates the use of the RCT Classifier and Cochrane Crowd, to identify the RCTs found in authors’ search results. Cochrane author teams conducting intervention reviews that incorporate RCTs can access this workflow via the Cochrane Register of Studies. Author teams wishing to use the Screen4Me workflow should liaise directly with the Cochrane Review Group’s Information Specialist. To date (January 2022), Screen4Me has been used in over 50 Cochrane intervention reviews. Workload reduction in terms of screening burden varies depending on the prevalence of RCTs in the domain area and the sensitivity of the searches conducted. A recent internal, as yet (February 2022) unpublished evaluation by one of the authors (AN-S) showed a mean reduction in screening workload of 51% (range 29% to 74%). More information on Screen4Me can be found in the Cochrane Information Specialists’ Handbook (Cochrane Information Specialist Support Team 2021d).

In addition to learning from large datasets such as those generated by Cochrane Crowd, it is also possible for machine learning models to learn how to apply eligibility criteria for individual reviews. This approach uses a process called ‘active learning’ and it is able to semi-automate study selection by continuously promoting records most likely to be relevant to the top of the results list (O'Mara-Eves et al 2015). It is difficult for authors to determine in advance when it is safe to stop screening and allow some records to be eliminated automatically without manual assessment. Recent work has suggested that this barrier is not insurmountable, and that it is possible to estimate how many relevant records remain to be found based on the sample already screened (Sneyd and Stevenson 2019, Callaghan and Muller-Hansen 2020, Li and Kanoulas 2020). The automatic elimination of records using this approach has not been recommended for use in Cochrane Reviews at the time of writing. This active learning process can still be useful, however, since by prioritizing records for screening in order of relevance, it enables authors to identify the studies that are most likely to be included much earlier in the screening process than would otherwise be possible.

Finally, tools are available that use natural language processing to highlight sentences and key phrases automatically (e.g. PICO elements, trial characteristics, details of randomization) to support the reviewer whilst screening (Tsafnat et al 2014).

4.7 Chapter information

Authors: Carol Lefebvre, Julie Glanville, Simon Briscoe, Robin Featherstone, Anne Littlewood, Chris Marshall, Maria-Inti Metzendorf, Anna Noel-Storr, Robin Paynter, Tamara Rader, James Thomas, L. Susan Wieland; on behalf of the Cochrane Information Retrieval Methods Group

Acknowledgements: This chapter has been developed from sections of previous editions of the Handbook co-authored since 1995 by Kay Dickersin, Julie Glanville, Kristen Larson, Carol Lefebvre, Eric Manheimer and Farhad Shokraneh. Many of the sources listed in this chapter and the accompanying online Technical Supplement have been brought to our attention by a variety of people over the years and we should like to acknowledge this. We should like to acknowledge: Ruth Foxlee, (formerly) Information Specialist, Cochrane Editorial Unit; Miranda Cumpston, (formerly) Head of Learning & Support, Cochrane Central Executive; Colleen Finley, Product Manager, John Wiley and Sons, for checking sections relating to searching the Cochrane Library; the (UK) National Institute for Health and Care Excellence and the German Institute for Quality and Efficiency in Health Care (IQWiG) for support in identifying some of the references; the (US) Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare Program Scientific Resource Center Article Alert service; Tianjing Li, Co-Convenor, Comparing Multiple Interventions Methods Group, for text and references that formed the basis of the re-drafting of parts of Section 4.6 Selecting studies; Lesley Gillespie, Cochrane author and former Editor and Trials Search Co-ordinator of the Cochrane Bone, Joint and Muscle Trauma Group, for copy-editing an early draft; the Cochrane Information Specialist Executive, the Cochrane Information Specialists’ Support Team, Cochrane Information Specialists and members of the Cochrane Information Retrieval Methods Group for comments on drafts; Su Golder, Co-Convenor, Adverse Effects Methods Group and Steve McDonald, Co-Director, Cochrane Australia for peer review.

4.8 References

Aamodt M, Huurdeman H, Strømme H. Librarian Co-Authored Systematic Reviews are Associated with Lower Risk of Bias Compared to Systematic Reviews with Acknowledgement of Librarians or No Participation by Librarians. Evidence Based Library and Information Practice 2019; 14: 103-127.

Agency for Healthcare Research and Quality. Methods guide for effectiveness and comparative effectiveness reviews: AHRQ publication no. 10(14)-EHC063-EF. 2014. https://effectivehealthcare.ahrq.gov/topics/cer-methods-guide/overview.

Arber M, Cikalo M, Glanville J, Lefebvre C, Varley D, Wood H. Annotated bibliography of published studies addressing searching for unpublished studies and obtaining access to unpublished data. 2013. https://methods.cochrane.org/sites/methods.cochrane.org.irmg/files/public/uploads/Annotatedbibliographtifyingunpublishedstudies.pdf.

Arber M, Wood H. Search strategy development [webpage]. Summarized Research in Information Retrieval for HTA (SuRe Info) 2009 Last updated 2021. https://sites.google.com/york.ac.uk/sureinfo/home.

Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Research Synthesis Methods 2015; 6: 87-95.

Bai Y, Gao J, Zou D, Li Z. Is MEDLINE alone enough for a meta-analysis? Alimentary Pharmacology and Therapeutics 2007; 26: 125-126; author reply 126.

Baudard M, Yavchitz A, Ravaud P, Perrodeau E, Boutron I. Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ 2017; 356: j448.

Bennett DA, Jull A. FDA: untapped source of unpublished trials. Lancet 2003; 361: 1402-1403.

Berber S, Tan-Koay AG, Opiyo N, Dwan K, Glanville JM, Lasserson TJ, Willson ML. A cross-sectional audit showed that most Cochrane intervention reviews searched trial registers. Journal of Clinical Epidemiology 2019; 113: 86-91.

Bero L. Searching for unpublished trials using trials registers and trials web sites and obtaining unpublished trial data and corresponding trial protocols from regulatory agencies. 2013. http://web.archive.org/web/20150108071243/http://methods.cochrane.org:80/projects-developments/searching-unpublished-trials-using-trials-registers-and-trials-web-sites-and-o

Booth A. How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. International Journal of Technology Assessment in Health Care 2010; 26: 431-435.

Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Systematic Reviews 2016; 5: 74.

Booth A, Briscoe S, Wright JM. The "realist search": A systematic scoping review of current practice and reporting. Research Synthesis Methods 2020; 11: 14-35.

Bramer WM, Giustini D, Kramer BM. Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study. Systematic Reviews 2016; 5: 39.

Bramer WM. Variation in number of hits for complex searches in Google Scholar. Journal of the Medical Library Association 2016; 104: 143-145.

Bramer WM, Milic J, Mast F. Reviewing retrieved references for inclusion in systematic reviews using EndNote. Journal of the Medical Library Association 2017; 105: 84-87.

Bravata DM, McDonald KM, Shojania KG, Sundaram V, Owens DK. Challenges in systematic reviews: synthesis of topics related to the delivery, organization, and financing of health care. Annals of Internal Medicine 2005; 142: 1056-1065.

Briscoe S. A review of the reporting of web searching to identify studies for Cochrane systematic reviews. Research Synthesis Methods 2018; 9: 89-99.

Brueton V, Tierney JF, Stenning S, Rait G. Identifying additional studies for a systematic review of retention strategies in randomised controlled trials: making contact with trials units and trial methodologists. Systematic Reviews 2017; 6: 167.

Callaghan MW, Muller-Hansen F. Statistical stopping criteria for automated screening in systematic reviews. Systematic Reviews 2020; 9: 273.

Callaway J. Librarian Reserve Corps: Literature indexing and metadata enhancement (LIME) observations from a year in the field. Journal of Health Information and Libraries Australasia 2021; 2: 35-45.

Center for Drug Evaluation and Research. Approval Package for EpiPen Auto Injector and EpiPen Jr. Auto Injector. 1989. Available from: https://www.accessdata.fda.gov/drugsatfda_docs/nda/pre96/019430Orig1s001.pdf.

Centre for Reviews and Dissemination. Systematic Reviews: CRD's guidance for undertaking reviews in health care. York: University of York; 2009.

Chan AW. Out of sight but not out of mind: how to search for unpublished clinical trial evidence. BMJ 2012; 344: d8013.

Chapman SJ, Shelton B, Mahmood H, Fitzgerald JE, Harrison EM, Bhangu A. Discontinuation and non-publication of surgical randomised controlled trials: observational study. BMJ 2014; 349: g6870.

Chilcott J, Brennan A, Booth A, Karnon J, Tappenden P. The role of modelling in prioritising and planning clinical trials. Health technology assessment (Winchester, England) 2003; 7: iii, 1-125.

Chow TK. Electronic search strategies should be repeatable. European Journal of Pain 2015; 19: 1562-1563.

Cochrane Information Specialist Support Team. Section 1: Role of a Cochrane Information Specialist. 2021a. In: Cochrane Information Specialists' Handbook [Internet]. Available from: https://training.cochrane.org/resource/cochrane-information-specialists-handbook.

Cochrane Information Specialist Support Team. Section 6: Author support. 2021b. In: Cochrane Information Specialists' Handbook [Internet]. Available from: https://training.cochrane.org/resource/cochrane-information-specialists-handbook.

Cochrane Information Specialist Support Team. Appendices. 2021c. In: Cochrane Information Specialists' Handbook [Internet]. Available from: https://training.cochrane.org/resource/cochrane-information-specialists-handbook.

Cochrane Information Specialist Support Team. Section 3: The Cochrane Register of Studies (CRS). 2021d. In: Cochrane Information Specialists' Handbook [Internet]. Available from: https://training.cochrane.org/resource/cochrane-information-specialists-handbook.

Committee on Publication Ethics (COPE) Council. COPE Guidelines: Retraction guidelines. Version 2: November. 2019. https://publicationethics.org/files/retraction-guidelines.pdf.

Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A, McIlroy W, Oxman AD. Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 1993; 269: 2749-2753.

Cooper C, Booth A, Varley-Campbell J, Britten N, Garside R. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Medical Research Methodology 2018a; 18: 85.

Cooper C, Dawson S, Peters J, Varley-Campbell J, Cockcroft E, Hendon J, Churchill R. Revisiting the need for a literature search narrative: A brief methodological note. Research Synthesis Methods 2018b; 9: 361-365.

Cooper C, Bou JT, Varley-Campbell J. Evaluating the effectiveness, efficiency, cost and value of contacting study authors in a systematic review: a case study and worked example. BMC Medical Research Methodology 2019; 19: 45.

Cooper C, Court R, Kotas E, Schauberger U. A technical review of three clinical trials register resources indicates where improvements to the search interfaces are needed. Research Synthesis Methods 2021a; 12: 384-393.

Cooper C, Lorenc T, Schauberger U. What you see depends on where you sit: The effect of geographical location on web-searching for systematic reviews: A case study. Research Synthesis Methods 2021b; 12: 557-570.

Cooper H, Ribble RG. Influences on the outcome of literature searches for integrative research reviews. Science Communication 1989; 10: 179-201.

Craven J, Levay P. Recording database searches for systematic reviews - What is the value of adding a narrative to peer-review checklists? A case study of NICE interventional procedures guidance. Evidence Based Library and Information Practice 2011; 6: 72-87.

de Vet H, Eisinga A, Riphagen I, Aertgeerts B, Pewsner D. Chapter 7: Searching for studies. In: Deeks J, Bossuyt P, Gatsonis C, editors. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 04 (updated September 2008): The Cochrane Collaboration; 2008. https://methods.cochrane.org/sites/methods.cochrane.org.sdt/files/public/uploads/Chapter07-Searching-%28September-2008%29.pdf.

Decullier E, Huot L, Maisonneuve H. What time-lag for a retraction search on PubMed? BMC Research Notes 2014; 7: 395.

Dellavalle RP, Hester EJ, Heilig LF, Drake AL, Kuntzman JW, Graber M, Schilling LM. Information science. Going, going, gone: lost Internet references. Science 2003; 302: 787-788.

Doshi P, Dickersin K, Healy D, Vedula SS, Jefferson T. Restoring invisible and abandoned trials: a call for people to publish the findings. BMJ 2013; 346: f2865.

Doust JA, Pietrzak E, Sanders S, Glasziou PP. Identifying studies for systematic reviews of diagnostic tests was difficult due to the poor sensitivity and precision of methodologic filters and the lack of information in the abstract. Journal of Clinical Epidemiology 2005; 58: 444-449.

Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991; 337: 867-872.

Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Statistics in Medicine 2002; 21: 1635-1640.

Egger M, Zellweger Z, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet 1997; 350: 326-329.

Egger M, Smith GD. Bias in location and selection of studies. BMJ 1998; 316: 61-66.

Egger M, Jüni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technology Assessment 2003; 7: 1-76.

Elia N, von Elm E, Chatagner A, Popping DM, Tramèr MR. How do authors of systematic reviews deal with research malpractice and misconduct in original studies? A cross-sectional analysis of systematic reviews and survey of their authors. BMJ Open 2016; 6: e010442.

Elsevier. Embase Classic Fact Sheet; 2020. https://www.elsevier.com/__data/assets/pdf_file/0010/1189387/FACT-SHEETEmbase-ClassicApril-2020.pdf.

Elsevier. Embase in 2021 Fact Sheet; 2021. https://www.elsevier.com/__data/assets/pdf_file/0005/58982/ELSV_14198_Embase-in-2021_Fact-Sheet.pdf.

Elsevier. Embase content coverage; 2022. https://www.elsevier.com/solutions/embase-biomedical-research/embase-coverage-and-content.

EUnetHTA JA3WP6B2-2 Authoring Team. Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness. Diemen (The Netherlands): EUnetHTA; 2019. https://www.eunethta.eu/.

European Food Safety Authority. Application of systematic review methodology to food and feed safety assessments to support decision making. EFSA Journal 2010; 8: 1637.

Eysenbach G, Trudel M. Going, going, still there: using the WebCite service to permanently archive cited web pages. Journal of Medical Internet Research 2005; 7: e60.

Ferrante di Ruffano L, Davenport C, Eisinga A, Hyde C, Deeks JJ. A capture-recapture analysis demonstrated that randomized controlled trials evaluating the impact of diagnostic tests on patient outcomes are rare. Journal of Clinical Epidemiology 2012; 65: 282-287.

Franco JVA, Garrote VL, Escobar Liquitay CM, Vietto V. Identification of problems in search strategies in Cochrane Reviews. Research Synthesis Methods 2018; 9: 408-416.

Frandsen TF, Gildberg FA, Tingleff EB. Searching for qualitative health research required several databases and alternative search strategies: a study of coverage in bibliographic databases. Journal of Clinical Epidemiology 2019a; 114: 118-124.

Frandsen TF, Eriksen MB, Hammer DMG, Christensen JB. PubMed coverage varied across specialties and over time: a large-scale study of included studies in Cochrane reviews. Journal of Clinical Epidemiology 2019b; 112: 59-66.

Frandsen TF, Bruun Nielsen MF, Lindhardt CL, Eriksen MB. Using the full PICO model as a search tool for systematic reviews resulted in lower recall for some PICO elements. Journal of Clinical Epidemiology 2020; 127: 69-75.

Fraser N, Brierley L, Dey G, Polka JK, Palfy M, Nanni F, Coates JA. The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape. PLoS Biology 2021; 19: e3000959.

Garritty C, Gartlehner G, Nussbaumer-Streit B, King VJ, Hamel C, Kamel C, Affengruber L, Stevens A. Cochrane Rapid Reviews Methods Group offers evidence-informed guidance to conduct rapid reviews. Journal of Clinical Epidemiology 2021; 130: 13-22.

Gartlehner G, Affengruber L, Titscher V, Noel-Storr A, Dooley G, Ballarini N, Konig F. Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial. Journal of Clinical Epidemiology 2020; 121: 20-28.

Gianola S, Jesus TS, Bargeri S, Castellini G. Characteristics of academic publications, preprints, and registered clinical trials on the COVID-19 pandemic. PLoS One 2020; 15: e0240123.

Glanville J, Lefebvre C, Wright K, (editors). ISSG search filter resource York (UK): The InterTASC Information Specialists' Sub-Group 2019. https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/home.

Glanville JM, Duffy S, McCool R, Varley D. Searching ClinicalTrials.gov and the International Clinical Trials Registry Platform to inform systematic reviews: what are the optimal search approaches? Journal of the Medical Library Association 2014; 102: 177-183.

Godard-Sebillotte C, Le Berre M, Karunananthan S, Hong QN, Vedel I. A digital media strategy to obtain unpublished data for a systematic review yields a very high author response rate. Journal of Clinical Epidemiology 2018; 104: 141-143.

Goldacre B, Lane S, Mahtani KR, Heneghan C, Onakpoya I, Bushfield I, Smeeth L. Pharmaceutical companies' policies on access to trial data, results, and methods: audit study. BMJ 2017; 358: j3334.

Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 2005; 331: 1064-1065.

Haddaway NR, Collins AM, Coughlin D, Kirk S. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PLoS One 2015; 10: e0138237.

Halfpenny NJ, Quigley JM, Thompson JC, Scott DA. Value and usability of unpublished data sources for systematic reviews and network meta-analyses. Evidence-based Medicine 2016; 21: 208-213.

Halladay CW, Trikalinos TA, Schmid IT, Schmid CH, Dahabreh IJ. Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions. Journal of Clinical Epidemiology 2015; 68: 1076-1084.

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Medical Research Methodology 2016; 16: 127.

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Medical Research Methodology 2017; 17: 64.

Hetherington J, Dickersin K, Chalmers I, Meinert CL. Retrospective and prospective identification of unpublished controlled trials: lessons from a survey of obstetricians and pediatricians. Pediatrics 1989; 84: 374-380.

Hinde S, Spackman E. Bidirectional citation searching to completion: an exploration of literature searching methods. Pharmacoeconomics 2015; 33: 5-11.

Horton R. Medical editors trial amnesty. Lancet 1997; 350: 756.

Huang X, Lin J, Demner-Fushman D. Evaluation of PICO as a knowledge representation for clinical questions. AMIA Annual Symposium Proceedings/AMIA Symposium 2006: 359-363.

Hwang TJ, Carpenter D, Lauffenburger JC, Wang B, Franklin JM, Kesselheim AS. Failure of investigational drugs in late-stage clinical development and publication of trial results. JAMA Internal Medicine 2016; 176: 1826-1833.

Iansavichene AE, Sampson M, McGowan J, Ajiferuke IS. Should systematic reviewers search for randomized, controlled trials published as letters? Annals of Internal Medicine 2008; 148: 714-715.

Institute for Quality and Efficiency in Health Care. IQWiG General Methods: Version 6.0. Cologne; 2020.

Institute of Medicine. Finding what works in health care: Standards for systematic reviews. Washington, DC: The National Academies Press; 2011. http://books.nap.edu/openbook.php?record_id=13059.

Irvin E, Hayden J. Developing and testing an optimal search strategy for identifying studies of prognosis [Poster] 14th Cochrane Colloquium, Dublin, Ireland; October 23-26 2006. https://abstracts.cochrane.org/2006-dublin/developing-and-testing-optimal-search-strategy-identifying-studies-prognosis.

Isojarvi J, Wood H, Lefebvre C, Glanville J. Challenges of identifying unpublished data from clinical trials: getting the best out of clinical trials registers and other novel sources. Research Synthesis Methods 2018: 561-578.

Jefferson T, Doshi P, Boutron I, Golder S, Heneghan C, Hodkinson A, Jones M, Lefebvre C, Stewart LA. When to include clinical study reports and regulatory documents in systematic reviews. BMJ Evidence-Based Medicine 2018; 23: 210-217.

Jiao S, Tsutani K, Haga N. Review of Cochrane reviews on acupuncture: how Chinese resources contribute to Cochrane reviews. Journal of Alternative and Complementary Medicine 2013; 19: 613-621.

Jüni P, Holenstein F, Sterne J, Bartlett C, Egger M. Direction and impact of language bias in meta-analyses of controlled trials: empirical study. International Journal of Epidemiology 2002; 31: 115-123.

Kastner M, Straus SE, McKibbon KA, Goldsmith CH. The capture-mark-recapture technique can be used as a stopping rule when searching in systematic reviews. Journal of Clinical Epidemiology 2009; 62: 149-157.

Kirkham JJ, Penfold NC, Murphy F, Boutron I, Ioannidis JP, Polka J, Moher D. Systematic examination of preprint platforms for use in the medical and biomedical sciences setting. BMJ Open 2020; 10: e041849.

Knelangen M, Hausner E, Metzendorf M-I, Sturtz S, Waffenschmidt S. Trial registry searches for randomized controlled trials of new drugs required registry-specific adaptation to achieve adequate sensitivity. Journal of Clinical Epidemiology 2018; 94: 69-75.

Köhler M, Haag S, Biester K, Brockhaus AC, McGauran N, Grouven U, Kölsch H, Seay U, Hörn H, Moritz G, Staeck K, Wieseler B. Information on new drugs at market entry: retrospective analysis of health technology assessment reports versus regulatory reports, journal publications, and registry reports. BMJ 2015; 350: h796.

Kreis J, Panteli D, Busse R. How health technology assessment agencies address the issue of unpublished data. International Journal of Technology Assessment in Health Care 2014; 30: 34-43.

Lampert A, Hoffmann GF, Ries M. Ten years after the International Committee of Medical Journal Editors' clinical trial registration initiative, one quarter of phase 3 pediatric epilepsy clinical trials still remain unpublished: a cross sectional analysis. PLoS One 2016; 11: e0144973.

Lane D, Dykeman J, Ferri M, Goldsmith CH, Stelfox HT. Capture-mark-recapture as a tool for estimating the number of articles available for systematic reviews in critical care medicine. Journal of Critical Care 2013; 28: 469-475.

Layton D. A critical review of search strategies used in recent systematic reviews published in selected prosthodontic and implant-related journals: Are systematic reviews actually systematic? International Journal of Prosthodontics 2017; 30: 13-21.

Lee K, Bacchetti P, Sim I. Publication of clinical trials supporting successful new drug applications: a literature analysis. PLoS Medicine 2008; 5: e191.

Lefebvre C, Eisinga A, McDonald S, Paul N. Enhancing access to reports of randomized trials published world-wide--the contribution of EMBASE records to the Cochrane Central Register of Controlled Trials (CENTRAL) in The Cochrane Library. Emerging Themes in Epidemiology 2008; 5: 13.

Lefebvre C, Glanville J, Wieland LS, Coles B, Weightman AL. Methodological developments in searching for studies for systematic reviews: past, present and future? Systematic Reviews 2013; 2: 78.

Lefebvre C, Duffy S. Peer review of searches for studies for health technology assessments, systematic reviews, and other evidence syntheses. International Journal of Technology Assessment in Health Care 2021; 37: e64.

Li D, Kanoulas E. When to Stop Reviewing in Technology-Assisted Reviews: Sampling from an Adaptive Distribution to Estimate Residual Relevant Documents. ACM Transactions on Information Systems 2020; 38: Article 41.

Lorenc T, Tyner EF, Petticrew M, Duffy S, Martineau FP, Phillips G, Lock K. Cultures of evidence across policy sectors: systematic review of qualitative evidence. European Journal of Public Health 2014; 24: 1041-1047.

Lorenzetti DL, Topfer LA, Dennett L, Clement F. Value of databases other than MEDLINE for rapid health technology assessments. International Journal of Technology Assessment in Health Care 2014; 30: 173-178.

MacLean CH, Morton SC, Ofman JJ, Roth EA, Shekelle PG, Southern California Evidence-Based Practice Center. How useful are unpublished data from the Food and Drug Administration in meta-analysis? Journal of Clinical Epidemiology 2003; 56: 44-51.

Manheimer E, Anderson D. Survey of public information about ongoing clinical trials funded by industry: evaluation of completeness and accessibility. BMJ 2002; 325: 528-531.

Marshall C, Brereton P. Systematic review toolbox: a catalogue of tools to support systematic reviews. Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering (EASE) 2015: Article no. 23.

Marshall I, Noel-Storr A, Kuiper J, Thomas J, Wallace BC. Machine learning for identifying randomized controlled trials: an evaluation and practitioner's guide. Research Synthesis Methods 2018; 9: 602-614.

McDonagh MS, Thakurta S, Peterson K. A study of the value of requesting information from drug manufacturers for systematic reviews; 9 years of experience from the drug effectiveness review project. Systematic Reviews 2018; 7: 172.

McGowan J, Sampson M, Salzwedel D, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Explanation and Elaboration (PRESS E&E) Ottawa: CADTH; 2016a. https://www.cadth.ca/sites/default/files/pdf/CP0015_PRESS_Update_Report_2016.pdf.

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. Journal of Clinical Epidemiology 2016b; 75: 40-46.

Meert D, Torabi N, Costella J. Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. Journal of the Medical Library Association 2016; 104: 267-277.

Metzendorf M-I. Why medical information specialists should routinely form part of teams producing high quality systematic reviews – a Cochrane perspective. Journal of the European Association for Health Information and Libraries 2016; 12: 6-9.

Metzendorf MI, Featherstone RM. Frandsen et al. provide insights to PubMed coverage across all Cochrane Review Groups, but more in-depth analyses would further help inform database choices. Journal of Clinical Epidemiology 2019; 116: 135-136.

Meursinge Reynders R, Ladu L, Di Girolamo N. Contacting of authors modified crucial outcomes of systematic reviews but was poorly reported, not systematic, and produced conflicting results. Journal of Clinical Epidemiology 2019; 115: 64-76.

Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health Technology Assessment 2003; 7: 1-90.

Morrison A, Polisena J, Husereau D, Moulton K, Clark M, Fiander M, Mierzwinski-Urban M, Clifford T, Hutton B, Rabb D. The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. International Journal of Technology Assessment in Health Care 2012; 28: 138-144.

Mullins MM, DeLuca JB, Crepaz N, Lyles CM. Reporting quality of search methods in systematic reviews of HIV behavioral interventions (2000–2010): are the searches clearly explained, systematic and reproducible? Research Synthesis Methods 2014; 5: 116-130.

National Institute for Health and Care Excellence. Chapter 5: Identifying the evidence: literature searching and evidence submission. Developing NICE guidelines: the manual. London: Process and methods [PMG20]; 2020.

Niederstadt C, Droste S. Reporting and presenting information retrieval processes: the need for optimizing common practice in health technology assessment. International Journal of Technology Assessment in Health Care 2010; 26: 450-457.

Noel-Storr A, Dooley G, Elliott J, Steele E, Shemilt I, Mavergames C, Wisniewski S, McDonald S, Murano M, Glanville J, Foxlee R, Beecher D, Ware J, Thomas J. An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials. Journal of Clinical Epidemiology 2021; 133: 130-139.

Noel-Storr AH, Dooley G, Wisniewski S, Glanville J, Thomas J, Cox S, Featherstone R, Foxlee R. Cochrane Centralised Search Service showed high sensitivity identifying randomized controlled trials: A retrospective analysis. Journal of Clinical Epidemiology 2020; 127: 142-150.

Nussbaumer-Streit B, Klerings I, Dobrescu AI, Persad E, Stevens A, Garritty C, Kamel C, Affengruber L, King VJ, Gartlehner G. Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study. Journal of Clinical Epidemiology 2020; 118: 42-54.

O'Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic Reviews 2015; 4: 5. (Erratum in: Systematic Reviews 2015; 2014: 2059).

Oikonomidi T, Boutron I, Pierre O, Cabanac G, Ravaud P, Consortium C-N. Changes in evidence for studies assessing interventions for COVID-19 reported in preprints: meta-research study. BMC Medicine 2020; 18: 402.

Oxman AD, Guyatt GH. The science of reviewing research. Annals of the New York Academy of Sciences 1993; 703: 125-133; discussion 133-134.

Page MJ, McKenzie JE, Higgins JPT. Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review. BMJ Open 2018; 8: e019703.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hrobjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021a; 372: n71.

Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hrobjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, McKenzie JE. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ 2021b; 372: n160.

Pansieri C, Pandolfini C, Bonati M. Clinical trial registries: more international, converging efforts are needed. Trials 2017; 18: 86.

Petticrew M, Roberts H. Systematic Reviews in the Social Sciences: a Practical Guide. Oxford (UK): Blackwell; 2006.

Pham B, Klassen TP, Lawson ML, Moher D. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. Journal of Clinical Epidemiology 2005; 58: 769-776.

Pieper D, Puljak L. Language restrictions in systematic reviews should not be imposed in the search strategy but in the eligibility criteria if necessary. Journal of Clinical Epidemiology 2021; 132: 146-147.

Pilkington K, Boshnakova A, Clarke M, Richardson J. "No language restrictions" in database searches: what does this really mean? Journal of Alternative and Complementary Medicine 2005; 11: 205-207.

Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Research Synthesis Methods 2014; 5: 98-115.

Relevo R, Paynter R. Peer Review of Search Strategies. Rockville (MD): Agency for Healthcare Research and Quality (US); 2012. https://www.ncbi.nlm.nih.gov/books/NBK98353/.

Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. Journal of Clinical Epidemiology 2015; 68: 617-626.

Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB, PRISMA-S. Group. PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. Systematic Reviews 2021; 10: 39.

Reveiz L, Cardona AF, Ospina EG, de Agular S. An e-mail survey identified unpublished studies for systematic reviews. Journal of Clinical Epidemiology 2006; 59: 755-758.

Rice DB, Kloda LA, Levis B, Qi B, Kingsland E, Thombs BD. Are MEDLINE searches sufficient for systematic reviews and meta-analyses of the diagnostic accuracy of depression screening tools? A review of meta-analyses. Journal of Psychosomatic Research 2016; 87: 7-13.

Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions. ACP Journal Club 1995; 123: A12-13.

Roundtree AK, Kallen MA, Lopez-Olivo MA, Kimmel B, Skidmore B, Ortiz Z, Cox V, Suarez-Almazor ME. Poor reporting of search strategy and conflict of interest in over 250 narrative and systematic reviews of two biologic agents in arthritis: a systematic review. Journal of Clinical Epidemiology 2009; 62: 128-137.

Royle P, Milne R. Literature searching for randomized controlled trials used in Cochrane reviews: rapid versus exhaustive searches. International Journal of Technology Assessment in Health Care 2003; 19: 591-603.

Royle P, Waugh N. Should systematic reviews include searches for published errata? Health Information & Libraries Journal 2004; 21: 14-20.

Salvador-Olivan JA, Marco-Cuenca G, Arquero-Aviles R. Errors in search strategies used in systematic reviews and their effects on information retrieval. Journal of the Medical Library Association 2019; 107: 210-221.

Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, St John PD, Viola R, Raina P. Should meta-analysts search Embase in addition to Medline? Journal of Clinical Epidemiology 2003; 56: 943-955.

Sampson M, Zhang L, Morrison A, Barrowman NJ, Clifford TJ, Platt RW, Klassen TP, Moher D. An alternative to the hand searching gold standard: Validating methodological search filters using relative recall. BMC Medical Research Methodology 2006; 6: 33.

Sampson M, McGowan J. Errors in search strategies were identified by type and frequency. Journal of Clinical Epidemiology 2006; 59: 1057-1063.

Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting methods for systematic reviews. Journal of Clinical Epidemiology 2008; 61: 748-754.

Sampson M, McGowan J. Inquisitio validus Index Medicus: A simple method of validating MEDLINE systematic review searches. Research Synthesis Methods 2011; 2: 103-109.

Sampson M, de Bruijn B, Urquhart C, Shojania K. Complementary approaches to searching MEDLINE may be sufficient for updating systematic reviews. Journal of Clinical Epidemiology 2016; 78: 108-115.

Scherer RW, Ugarte-Gil C, Schmucker C, Meerpohl JJ. Authors report lack of time as main reason for unpublished research presented at biomedical conferences: a systematic review. Journal of Clinical Epidemiology 2015; 68: 803-810.

Schroll JB, Bero L, Gøtzsche PC. Searching for unpublished data for Cochrane reviews: cross sectional study. BMJ 2013; 346: f2231.

Shemilt I, Simon A, Hollands GJ, Marteau TM, Ogilvie D, O'Mara-Eves A, Kelly MP, Thomas J. Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Research Synthesis Methods 2014; 5: 31-49.

Shemilt I, Khan N, Park S, Thomas J. Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Systematic Reviews 2016; 5: 140.

Sneyd A, Stevenson M. Modelling Stopping Criteria for Search Results using Poisson Processes. 2019. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) [Internet]. [3484–3489]. Available from: https://aclanthology.org/D19-1351.

Spencer AJ, Eldredge JD. Roles for librarians in systematic reviews: a scoping review. Journal of the Medical Library Association 2018; 106: 46-56.

Spoor P, Airey M, Bennett C, Greensill J, Williams R. Use of the capture-recapture technique to evaluate the completeness of systematic literature searches. BMJ 1996; 313: 342-343.

Spry C, Mierzwinski-Urban M, Rabb D. Peer review of literature search strategies: does it make a difference? 21st Cochrane Colloquium; 2013; Quebec City, Canada. https://abstracts.cochrane.org/2013-quebec-city/peer-review-literature-search-strategies-does-it-make-difference.

Spry C, Mierzwinski-Urban M. The impact of the peer review of literature search strategies in support of rapid review reports. Research Synthesis Methods 2018; 9: 521-526.

Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complementary Therapies in Medicine 2004; 12: 228-232.

Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Controlled Clinical Trials 2000; 21: 476-487.

Thomas J, Noel-Storr A, Marshall I, Wallace B, McDonald S, Mavergames C, Glasziou P, Shemilt I, Synnot A, Turner T, Elliott J, Living Systematic Review N. Living systematic reviews: 2. Combining human and machine effort. Journal of Clinical Epidemiology 2017; 91: 31-37.

Thomas J, McDonald S, Noel-Storr A, Shemilt I, Elliott J, Mavergames C, Marshall IJ. Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews. Journal of Clinical Epidemiology 2021; 133: 140-151.

Tramèr MR, Reynolds DJ, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis: a case study. BMJ 1997; 315: 635-640.

Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Systematic Reviews 2014; 3: 74.

US National Library of Medicine. MEDLINE, PubMed, and PMC (PubMed Central): How are they different? 2020. https://www.nlm.nih.gov/bsd/difference.html.

US National Library of Medicine. MEDLINE: Overview 2021. https://www.nlm.nih.gov/medline/medline_overview.html.

US National Library of Medicine. PubMed Overview no date. https://pubmed.ncbi.nlm.nih.gov/about/.

Viergever RF, Li K. Trends in global clinical trial registration: an analysis of numbers of registered clinical trials in different parts of the world from 2004 to 2013. BMJ Open 2015; 5: e008932.

von Elm E, Poglia G, Walder B, Tramèr MR. Different patterns of duplicate publication: an analysis of articles used in systematic reviews. JAMA 2004; 291: 974-980.

Waffenschmidt S, Knelangen M, Sieben W, Buhn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Medical Research Methodology 2019; 19: 132.

Wallace S, Daly C, Campbell M, Cody J, Grant A, Vale L, Donaldson C, Khan I, Lawrence P, MacLeod A. After MEDLINE? Dividend from other potential sources of randomised controlled trials. Second International Conference Scientific Basis of Health Services & Fifth Annual Cochrane Colloquium; 1997; Amsterdam, The Netherlands.

Weber EJ, Callaham ML, Wears RL, Barton C, Young G. Unpublished research from a medical specialty meeting: why investigators fail to publish. JAMA 1998; 280: 257-259.

Wieseler B, Kerekes MF, Vervoelgyi V, McGauran N, Kaiser T. Impact of document type on reporting quality of clinical drug trials: a comparison of registry reports, clinical study reports, and journal publications. BMJ 2012; 344: d8141.

Witkowski MA, Aldhouse N. Transparency and reproducibility of supplementary search methods in NICE single technology appraisal manufacturer submissions. Value in Health 2015; 18: A721-722.

Woods D, Trewheellar K. Medline and Embase complement each other in literature searches. BMJ 1998; 316: 1166.

Wright K, McDaid C. Reporting of article retractions in bibliographic databases and online journals. Journal of the Medical Library Association 2011; 99: 164-167.

Wu XY, Tang JL, Mao C, Yuan JQ, Qin Y, Chung VC. Systematic reviews and meta-analyses of traditional Chinese medicine must search Chinese databases to reduce language bias. Evidence-Based Complementary and Alternative Medicine 2013; 2013: 812179.

Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association 2009; 97: 21-29.

Young T, Hopewell S. Methods for obtaining unpublished data. Cochrane Database of Systematic Reviews 2011; 11: MR000027.

Ziel J. Email communication between Josephine Ziel (Product Manager, Content Assets, Biomedical & Biology, Elsevier Life Sciences Solutions Biology Products) and Carol Lefebvre (chapter author). 2022.

Zwakman M, Verberne LM, Kars MC, Hooft L, van Delden JJM, Spijker R. Introducing PALETTE: an iterative method for conducting a literature search for a review in palliative care. BMC Palliative Care 2018; 17: 82.