Carol Lefebvre, Julie Glanville, Simon Briscoe, Robin Featherstone, Anne Littlewood, Maria-Inti Metzendorf, Anna Noel-Storr, Robin Paynter, Tamara Rader, James Thomas, L. Susan Wieland; on behalf of the Cochrane Information Retrieval Methods Group

Cite this chapter as: Lefebvre C, Glanville J, Briscoe S, Featherstone R, Littlewood A, Metzendorf M-I, Noel-Storr A, Paynter R, Rader T, Thomas J, Wieland LS. Chapter 4: Searching for and selecting studies [last updated September 2024]. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane, 2024. Available from www.training.cochrane.org/handbook.

4.1 Introduction

Cochrane Reviews take a systematic and comprehensive approach to identifying studies that meet the eligibility criteria for the review. This chapter outlines some general issues in searching for and selecting studies. It describes the main sources of potential studies; and discusses how to plan the search process, design and carry out search strategies, manage referencesand other records found during the search process, correctly document the search process and select studies from the search results.

This chapter aims to provide review authors with background information on all aspects of searching for and selecting studies so that they can better understand the search and selection processes. All authors of systematic reviews should, however, identify an experienced medical/healthcare librarian or information specialist to collaborate with on the search process. The chapter also aims to provide advice and guidance for medical/healthcare librarians and information specialists (within and beyond Cochrane) involved in the search process to identify studies for inclusion in systematic reviews.

This chapter focuses on searching for randomized trials for systematic reviews of interventions. Many of the search principles discussed, however, will also apply to other study designs and review types. This Handbook (for systematic reviews of interventions) contains other chapters which should also be considered. For example, searching for non-randomized studies is discussed in Chapter 24 (see also Chapter 19 for adverse effects). Other discussion of searching for specific types of evidence appears in chapters dedicated to these types of evidence, such as Chapter 17 on intervention complexity, Chapter 20 on economic evidence and Chapter 21 on qualitative evidence. Furthermore, additional Cochrane Handbooks are in various stages of development, for example diagnostic test accuracy studies (published)(Spijker et al 2023), qualitative evidence (in draft) (Stansfield et al 2024) and prognosis studies (under development).

An online Technical Supplement to this chapter provides more detail on searching methods.

4.2 General issues

4.2.1 Role of the information specialist/librarian

Medical/healthcare librarians and information specialists have an integral role in the production of Cochrane Reviews. There is increasing evidence of the involvement of information specialists in systematic reviews (Spencer and Eldredge 2018, Ross-White 2021, Schvaneveldt and Stellrecht 2021, Brunskill and Hanneke 2022, Lê et al 2023 ) and evidence to support the improvement in the quality of various aspects of the search process (Koffel 2015, Rethlefsen et al 2015, Meert et al 2016, Metzendorf 2016, Aamodt et al 2019, Hameed et al 2020, Schellinger et al 2021, Ghezzi-Kopel et al 2022, Ramirez et al 2022, Pawliuk et al 2024, Giroudon et al 2024).

Many Cochrane Groups employ an information specialist to collaborate with authors on the search process. The range of services, however, offered by Cochrane Groups and/or their information specialists varies according to the resources available. Cochrane Review authors should, therefore, contact their Cochrane Group or the Central Editorial Service at the earliest stage to find out what advice and level of contribution is available to them. Authors conducting their own searches should seek advice from a Cochrane Information Specialist not only on which sources to search, but also with respect to the exact strategies to be run (see Section 4.4). If the Cochrane Group does not provide this service or employ an information specialist, we recommend that review authors initially seek guidance from Cochrane Support (support@cochrane.org). Otherwise, authors should seek help with their searches from a medical/healthcare librarian or information specialist, preferably one with experience in identifying studies for systematic reviews.

Cochrane Information Specialists are responsible for working with authors in searching for studies for inclusion in their reviews, and for keeping up to date with Cochrane methodological developments in information retrieval (Cochrane Information Specialist Support Team 2021b). Some Cochrane Information Specialists maintain a Specialized Register for their Group, containing reports of trials relating to the Group’s scope. Within the limits of licensing restrictions, the content of these Group registers is shared with users worldwide via the Cochrane Central Register of Controlled Trials (CENTRAL), part of the Cochrane Library (see Section 4.3.1.3).

Most Cochrane Groups collaborate with authors in study identification from the early planning stage to the final write-up of the review and any updates. This may include some or all of the following:

• advising authors on which databases and other sources should be searched for their review;
• designing, or providing guidance on designing, search strategies for the main bibliographic databases and trials registers resources;
• running searches in databases and trials registers resources available to the information specialist;
• saving and collating search results, and sharing them with authors in appropriate formats;
• drafting, or working with authors in drafting, the search methods sections of a Cochrane protocol, review and/or update;
• advising authors on how to run searches in sources other than the main bibliographic databases and trials register resources and how to download results from these sources;
• ensuring that the conduct of Cochrane protocols, reviews and updates meets the requirements set out in the Methodological Expectations of Cochrane Intervention Reviews (MECIR) relating to searching activities for reviews, and that the reporting aligns with the current reporting guidance for PRISMA (Page et al 2021b, Page et al 2021a) and PRISMA-S (Rethlefsen et al 2021);
• organizing translations, or at least organizing data extraction, where required, from studies reported in languages other than English to enable authors to assess these reports for inclusion/exclusion in their reviews;
• obtaining copies of trial reports for review teams when required (within copyright legislation);
• providing advice to and collaborating with author teams on the use of reference management tools and other software used in review production, including review production tools such as Covidence, EPPI-Reviewer and RevMan; and
• checking and formatting the references to included and/or excluded studies in line with the Cochrane Style Manual.

4.2.2 Minimizing bias

Systematic reviews require a thorough, objective and reproducible search of a range of sources to identify as many eligible studies as possible (within resource limits). This is a major factor distinguishing systematic reviews from traditional narrative reviews, which helps to minimize bias and achieve more reliable estimates of effects and uncertainties. A search of MEDLINE alone is not considered adequate. Research evidence indicates that not all known published randomized trials are available in MEDLINE and that even if relevant records are in MEDLINE, it can be difficult to retrieve them (see Section 4.3.1.2).

Searching beyond MEDLINE is important not only for ensuring that as many relevant studies as possible are identified, but also to minimize selection bias for those that are found. Relying exclusively on a MEDLINE search may retrieve a set of reports unrepresentative of all reports that would have been identified through a wider or more extensive search of several sources.

Time and budget restraints require the review team to balance the thoroughness of the search with efficiency in the use of time and funds. The best way of achieving this balance is to be aware of, and try to minimize, the biases such as publication bias and language bias that can result from restricting searches in different ways (see Chapter 8 and Chapter 13 for further guidance on assessing these biases). Unlike for tasks such as study selection or data extraction, it is not considered necessary (or even desirable) for two people to conduct independent searches in parallel. It is strongly recommended, however, that all search strategies should be peer reviewed, before being run, by a suitably qualified and experienced medical/healthcare librarian or information specialist (see Section 4.4.8).

4.2.3 Studies versus reports of studies

Systematic reviews have studies as the primary units of interest and analysis. A single study may have more than one report about it (or record for it), and each of these reports or other records may contribute useful information for the review (see Section 4.6.1). For most of the sources listed in Section 4.3, the search process will retrieve individual reports of studies, so that multiple reports of the same study will need to be identified and associated with each other manually by the review authors. There is, however, an increasing number of study-based sources, which link multiple records of the same study together, such as the Cochrane Register of Studies and the Specialized Registers of a number of Cochrane Groups (see online Technical Supplement), and some other trials registers, regulatory and industry sources. Processes and software to select and group publications by study are discussed in Section 4.6.

4.2.4 Copyright and licensing

All review authors and others involved in Cochrane should adhere to copyright legislation and the terms of licensing agreements for databases and other resources. With respect to searching for studies, this refers in particular to adhering to the terms and conditions of use when searching databases and other resources and downloading records, as well as adhering to copyright legislation when obtaining copies of publications and other records. Review authors should seek guidance on this from a medical/healthcare librarian or information specialist, as copyright legislation varies across jurisdictions and licensing agreements vary across organizations.

4.3 Sources to search

The sections that follow refer to sources to search for studies for inclusion in intervention reviews, irrespective of the intervention. For more detailed discussion of specific issues around searching for medical devices, please refer to this recent method note (Cooper et al 2022a).

4.3.1 Bibliographic databases

4.3.1.1 Introduction to bibliographic databases

For further details on this topic, please refer to Section 1.1. of the online Technical Supplement and its subsections.

The search for studies for a Cochrane Review should be as extensive as possible in order to reduce the risk of reporting bias and to identify as much relevant evidence as possible (see MECIR Box 4.3.a). Searches of health-related bibliographic databases are generally the most efficient way to identify an initial set of relevant reports of studies (EUnetHTA JA3WP6B2-2 Authoring Team 2019). Database selection should be guided by the review topic (Suarez-Almazor et al 2000, Stevinson and Lawlor 2004, Lorenzetti et al 2014). Searching two or more databases lowers the risk of missing eligible studies (Ewald et al 2022, Boberg-Ans et al 2023). Especially when topics are specialized, cross-disciplinary, or involve emerging technologies (Rice et al 2016), additional databases may need to be identified and searched (Wallace et al 1997, Stevinson and Lawlor 2004, Frandsen et al 2019a). Methods for determining the coverage and comprehensiveness of bibliographic databases can help authors select appropriate sources for their review topic (Brody et al 2023).

MECIR Box 4.3.a Relevant expectations for conduct of intervention reviews

The three bibliographic databases generally considered to be the most important sources to search for reports of trials are CENTRAL (Noel-Storr et al 2020), MEDLINE (Halladay et al 2015, Sampson et al 2016) and Embase (Woods and Trewheellar 1998, Sampson et al 2003, Bai et al 2007). These databases are described in more detail in Sections 4.3.1.2 and 4.3.1.3 and in the online Technical Supplement. For Cochrane Reviews, CENTRAL, MEDLINE and Embase (if access to Embase is available to the review team) should be searched (see MECIR Box 4.3.a). These searches may be undertaken specifically for the review, or indirectly by searching the Cochrane Group’s Specialized Register, if it exists and was designed to support reviews in this way.

Some bibliographic databases, such as MEDLINE and Embase, include abstracts for the majority of recent records. A key advantage of such databases is that they can be searched electronically both for words in the title or abstract and by using the standardized indexing terms, or controlled vocabulary, assigned to each record (see Section 4.3.1.2 and 4.4.4). In addition to MEDLINE and Embase, Cochrane has developed a database of reports of randomized trials called the Cochrane Central Register of Controlled Trials (CENTRAL), which is published within the Cochrane Library (see Section 4.3.1.3).

Bibliographic databases are available to individuals for a fee (by subscription or on a ‘pay-as-you-go’ basis) or free at the point of use. They may be available through national provisions, site-wide licences at institutions such as universities or hospitals, through professional organizations as part of their membership packages or free-of-charge on the internet. Some international initiatives provide free or low-cost online access to databases (and full-text journals) over the internet. The Health InterNetwork Access to Research Initiative (HINARI) programme, set up by the World Health Organization (WHO) together with major publishers and now part of the Research4Life programme (R4L), provides access to a wide range of databases including the Cochrane Library for healthcare professionals in local, not-for-profit institutions in more than 120 countries, areas and territories. The International Network for the Availability of Scientific Publications (INASP) also provides access to a wide range of databases (and journals) including the Cochrane Library. Electronic Information for Libraries (EIFL) is a similar initiative based on library consortia to support affordable licensing of journals and other sources in more than 50 developing and transition countries in Africa, Asia, Europe and Latin America.

The online Technical Supplement provides more detailed information about how to search these sources and other databases. The accompanying Appendix provides a list of general healthcare databases by region and healthcare databases by subject area. Further evidence-based information about sources to search can be found on the SuRe Info portal, which is updated twice per year (Isojarvi and Glanville 2021).

4.3.1.2 MEDLINE and Embase

Cochrane Reviews of interventions should include a search of MEDLINE (see MECIR Box 4.3.a). MEDLINE (as of February 2024) contains approximately 30 million references to journal articles in biomedicine and health from 1946 onwards. More than 5,000 journals in about 40 languages are indexed for MEDLINE (US National Library of Medicine).

PubMed provides access to a free version of MEDLINE that also includes up-to-date citations not yet indexed for MEDLINE (US National Library of Medicine 2023). Additionally, PubMed includes records from journals that are not indexed for MEDLINE and records considered ‘out-of-scope’ from journals that are partially indexed for MEDLINE. Further details about MEDLINE, PubMed and PubMed Central and how they differ are available (US National Library of Medicine).

MEDLINE is also available on subscription from a number of other database vendors/platforms, such as EBSCO, Ovid, ProQuest, STN and Clarivate’s Web of Science. Access is usually ‘free at-the-point-of-use’ to members of the institutions paying the subscriptions (e.g. hospitals and universities). Ovid MEDLINE (segment name ‘MEDALL’) covers all of the available content and metadata in PubMed with a delay of one working day (except during the annual reload, at the end of each year, when Ovid MEDLINE will not match the PubMed baseline). Aside from the MEDLINE records, Ovid includes all content types available in PubMed including: Epub Ahead of Print, In-Process, In-Data-Review & Other Non-Indexed Citations.

When searching MEDLINE via service providers or interfaces other than Ovid or PubMed, we recommend verification of the exact coverage of the database in relation to PubMed, where no explicit information on this is readily available. Note that MEDLINE (i.e. PubMed) is searched regularly by Cochrane for reports of trials. These records are included in CENTRAL (see online Technical Supplement).

Cochrane Reviews of interventions should include a search of Embase (if access to Embase is available to the review team) (see MECIR Box 4.3.a). Embase (as of February 2024) contains more than 40 million records from 1947 onwards, including records from more than 8,000 currently published journals from approximately 100 countries (Elsevier 2024a). Embase now includes all MEDLINE (i.e. PubMed) records, thus, technically, allowing both databases to be searched simultaneously. Further details on the implications of this for searching are available in the online Technical Supplement. There are more than 10 million records in Embase from approximately 3,000 journals that are not indexed in MEDLINE (Elsevier 2024a). Embase Classic provides access to almost two million records digitized from the Excerpta Medica print journals (the original print indexes from which Embase was created) from 1947 to 1973 (Elsevier 2024a, Elsevier 2024b). Embase Classic is only available as an add-on to an Embase subscription. Embase also includes pre-print articles from medRxiv and bioRxiv; see Embase Release Notes November 2021.

Embase is only available by subscription, either directly via Elsevier (as Embase.com) or from other database vendors such as Ovid, ProQuest or STN. It is mandatory for Cochrane intervention reviews to include a search of Embase if access is available to the review team (see MECIR Box 4.3.a). Note that Embase is searched regularly by Cochrane for reports of trials. These records are included in CENTRAL (see online Technical Supplement).

The online Technical Supplement provides guidance on how to search MEDLINE and Embase for reports of trials. The actual degree of reference overlap between MEDLINE and Embase varies widely according to the topic, but studies comparing searches of the two databases have generally concluded that a comprehensive search requires that both databases be searched (Lefebvre et al 2008, Bramer et al 2016) (see MECIR Box 4.3.a).

Conversely, two studies examined different samples of Cochrane Reviews and identified the databases from which the included studies of these reviews originated (Halladay et al 2015, Hartling et al 2016). Halladay showed that the majority of included studies could be identified via PubMed (range 75% to 92%) and Hartling showed that the majority of included studies could be identified by using a combination of two databases, but the two databases were different in each case. Both studies, one across all healthcare areas (Halladay et al 2015) and the other on child health (Hartling et al 2016), report a minimal extent to which the inclusion of studies not indexed in PubMed altered the meta-analyses. PubMed coverage across systematic review topics has been further evaluated in a study based on a comprehensive sample of Cochrane Reviews. It provides further evidence of PubMed’s relatively high coverage (range 68% to 73%), with an emphasis that it is markedly variable across and within specialties (Frandsen et al 2019b, Metzendorf and Featherstone 2019). A study evaluating abbreviated literature searches suggested that a combination of MEDLINE and CENTRAL was sufficient to find the majority of available RCTs on clinical interventions. The authors concluded that, if decision-makers are willing to accept less certainty and a small risk for opposite conclusions, some abbreviated searches are viable options for rapid evidence syntheses. They also concluded, however, that decisions demanding high certainty require comprehensive searches (Nussbaumer-Streit et al 2018). Uncertainty remains about the circumstances under which it is most important to search multiple databases. Comprehensive searches of multiple databases may be more important for public health interventions (Levay et al 2022) and for study designs other than RCTs. The current recommendation of searching multiple databases, therefore, needs to be evaluated further, in order to confirm under which circumstances comprehensive searches of multiple databases are warranted.

4.3.1.3 The Cochrane Central Register of Controlled Trials (CENTRAL)

Since its inception, the Cochrane Central Register of Controlled Trials (CENTRAL) has been recognized as the most comprehensive source of reports of randomized trials (Egger and Smith 1998). Two recent studies have reconfirmed the high sensitivity of CENTRAL in identifying randomized controlled trials (Noel-Storr et al 2020, Noel-Storr and Wisniewski 2024). CENTRAL is published as part of the Cochrane Library and is updated monthly. As of April 2024, CENTRAL contains more than 2,000,000 records of reports of trials/trials registry records potentially eligible for inclusion in Cochrane Reviews, by far the majority of which are randomized trials (Noel-Storr et al 2020).

Many of the records in CENTRAL have been identified through systematic searches of MEDLINE, Embase, CINAHL Plus, the Australasian Medical Index, KoreaMed, ClinicalTrials.gov and the trial records available through the WHO International Clinical Trials Registry Portal (see online Technical Supplement). CENTRAL, however, also includes citations to reports of randomized trials that are not included in MEDLINE, Embase or other bibliographic databases; citations published in many languages; and citations that are available only in conference proceedings or other sources that are difficult to access. It also includes records from trials registers and trials results registers beyond ClinicalTrials.gov and the WHO portal.

Cochrane Information Specialists have previously conducted comprehensive searches to populate Specialized Registers in their field. The results of these searches were then added to CENTRAL. Only some Cochrane Groups, however, continue to maintain a Specialized Register. For the Cochrane Groups that no longer maintain a Specialized Register, its role has been replaced by the Cochrane Centralised Search Service (Noel-Storr et al 2020). Legacy records from the Specialized Registers that are no longer maintained continue to be available via CENTRAL. Where a Specialized Register is still available, for which sufficiently comprehensive searching has been conducted, and kept up-to-date, a search of the Specialized Register may be conducted instead of separately searching CENTRAL, MEDLINE and Embase for a specific review (but refer to Section 2.2 of the Technical Supplement to this chapter for caveats around this). In these cases, the search will be more precise but the majority of included studies will be identified with lower numbers of records to screen. There will, however, be a time lag between records appearing in databases such as MEDLINE or Embase and their inclusion in a Specialized Register.

CENTRAL is available through the Cochrane Library. Many review authors have full access free-of-charge at the point-of-use through national provisions and other similar arrangements, or as part of a paid subscription to the Cochrane Library. All Cochrane Information Specialists have full access to CENTRAL.

The online Technical Supplement provides information on what is in CENTRAL from MEDLINE, Embase and other sources, as well as guidance on searching CENTRAL.

4.3.1.4 Other bibliographic databases

For further details on this topic, please refer to Section 1.1 of the online Technical Supplement and its subsections.

Many countries and regions produce bibliographic databases that focus on the literature produced in those regions and which often include journals and other literature not indexed elsewhere. There are also subject-specific bibliographic databases, such as AMED (allied and complementary medicine), CINAHL (nursing and allied health) and APA PsycInfo (psychology and psychiatry). It is highly desirable that searches be conducted of appropriate national, regional and subject specific bibliographic databases (see MECIR Box 4.3.b). Further details are provided in the online Technical Supplement.

Citation indexes are bibliographic databases that record instances where a particular reference is cited, in addition to the standard bibliographic content. Citation indexes can be used to identify studies that are similar to a study report of interest, as it is probable that other reports citing or cited by a study will contain similar or related content. Further details are provided in the online Technical Supplement.

MECIR Box 4.3.b Relevant expectations for conduct of intervention reviews

4.3.2 Ongoing studies and unpublished data sources

For further details on this topic, please refer to Section 1.2 of the online Technical Supplement and its subsections.

Initiatives to provide access to ongoing studies and unpublished data constitute a fast-moving field (Isojarvi et al 2018). Review authors should therefore consult a medical/healthcare librarian or information specialist for current advice.

It is important to identify ongoing studies, so that when a review is updated these can be assessed for possible inclusion. Awareness of the existence of a possibly relevant ongoing study and its expected completion date might affect not only decisions with respect to when to update a specific review, but also when to aim to complete a review. Information about possibly relevant ongoing studies should be included in the review in the ‘Characteristics of ongoing studies’ table.

Even when studies are completed or terminated, some are never published. An association between ‘statistically significant’ results and publication has been documented across a number of studies, as summarized in Chapter 13. Finding out about unpublished studies, and including their results in a systematic review when eligible and appropriate (Cook et al 1993), is important for minimizing bias. Several studies and other articles addressing issues around identifying unpublished studies have been published (Easterbrook et al 1991, Weber et al 1998, Manheimer and Anderson 2002, MacLean et al 2003, Lee et al 2008, Chan 2012, Schroll et al 2013, Wolfe et al 2013, Chapman et al 2014, Kreis et al 2014, Scherer et al 2015, Hwang et al 2016, Iacobucci 2016, Lampert et al 2016, Campbell et al 2022, Speich et al 2022).

There is no easy and reliable single way to obtain information about studies that have been completed or terminated but never published. There have, however, been several important initiatives resulting in better access to studies and their results from sources other than the main bibliographic databases and journals. These include the further development of trials registers and trials results registers (see Section 4.3.3), and improved access to regulatory agency sources and clinical study reports (CSRs), which are the very detailed reports prepared by industry for regulatory approval (Egilman et al 2021) (see Section 4.3.4). A study (Halfpenny et al 2016) assessed the value and usability for systematic reviews and network meta-analyses of data from trials registers, CSRs and regulatory authorities, and concluded that data from these sources have the potential to influence systematic review results. Two earlier studies showed that a considerably higher proportion of CSRs prepared for regulatory approval of drugs provided complete information on study methods and results than did trials register records or journal publications (Wieseler et al 2012) and that conventional, publicly available sources (European Public Assessment Reports, journal publications, and trials register records) provide insufficient information on new drugs, especially on patient relevant outcomes in approved subpopulations (Köhler et al 2015).

An annotated bibliography of published studies addressing searching for unpublished studies and obtaining access to unpublished data is also available (Arber et al 2013). One particular study focused on the contribution of unpublished studies, including dissertations, and studies in languages other than English, to the results of meta-analyses in reviews relevant to children (Hartling et al 2017). They found that, in their sample, unpublished studies and studies in languages other than English rarely had any impact on the results and conclusions of the review. They did, however, concede that inclusion of these study types may have an impact in situations where there are few relevant studies, or where there are ‘questionable vested interests’ in the published literature.

Asking researchers for information about completed or terminated but never published studies has not always been found to be fruitful (Hetherington et al 1989, Horton 1997) though some researchers have reported that this is an important method for retrieving studies for systematic reviews (Royle and Milne 2003, Greenhalgh and Peacock 2005, Reveiz et al 2006, Godolphin et al 2019). Correspondence can be an important source of information about unpublished studies. It is highly desirable for authors of Cochrane Reviews of interventions to contact relevant individuals and organizations for information about unpublished or ongoing studies (see MECIR Box 4.3.c). Letters of request for information can be used to identify completed or terminated but unpublished studies. One way of doing this is to send a comprehensive list of relevant articles and study records along with the eligibility criteria for the review to the first author of reports of included studies, asking if they know of any additional studies (ongoing, completed or terminated; published or unpublished) that might be relevant. This approach may be especially useful in areas where there are few trials or a limited number of active research groups. It may also be desirable to send the same letter to other experts and pharmaceutical companies and others with an interest in the area. Some review teams set up websites for systematic review projects, listing the studies identified to date and inviting submission of information on studies not already listed. A Cochrane Methodology Review examined studies assessing methods for obtaining unpublished data and concluded that those carrying out systematic reviews should continue to contact authors for missing data and that email contact was more successful than other methods (Young and Hopewell 2011). A study assessed the value of contacting trial authors and concluded that data supplied by authors modified the outcomes of some systematic reviews, but this was poorly reported in the reviews (Meursinge Reynders et al 2019). A further case study evaluated the effectiveness, efficiency, cost and value of contacting study authors in a systematic review and concluded that this was cost-effective in terms of time taken and costs in carrying out this work compared with unique data identified from the authors’ replies (Cooper et al 2019). Another case study of a Cochrane Methodology Review reported that making contact with clinical trials units and trial methodologists provided data for six of the 38 RCTs included in the review, which had not been identified through other search methods (Brueton et al 2017). A further study reported successful outcomes of a digital media strategy to obtain unpublished data from trial authors by contacting them using a combination of email, ResearchGate and LinkedIn (Godard-Sebillotte et al 2018). A study assessed the value of requesting information from drug manufacturers for systematic reviews and concluded that this helped to reduce reporting and publication bias and helped to fill important gaps, sometimes leading to new or altered conclusions, primarily where no other evidence existed (McDonagh et al 2018).

MECIR Box 4.3.c Relevant expectations for conduct of intervention reviews

The RIAT (Restoring Invisible and Abandoned Trials) initiative (Doshi et al 2013) aims to address the problems outlined above by offering a methodology that allows others to re-publish mis-reported and to publish unreported trials. Anyone who can access the trial data and document trial abandonment can use this methodology. The RIAT Support Center offers free-of-charge support and competitive funding to researchers interested in this approach. It has also been suggested that legislation such as Freedom of Information Acts (FOIAs) in various countries might be used to gain access to information about unpublished trials (Bennett and Jull 2003, MacLean et al 2003, Coyne 2012, Doshi and Jefferson 2016, Yale Law School 2017). In the US, the FOIA is frequently used to obtain information about the US Department of Health and Human Services (HHS) and its agencies. It has been suggested, in light of growing costs, minimal fees collected and lengthy processing times, that HHS agencies’ FOIA programmes might be made more efficient through greater proactive record disclosure (Egilman et al 2019).

4.3.3 Trials registers and trials results registers

For further details on this topic, please refer to Section 1.2.1 of the online Technical Supplement.

Cochrane Reviews of interventions should search relevant trials registers and repositories of results (see MECIR Box 4.3.d). A recent audit by Cochrane investigators showed that the majority of Cochrane Reviews do comply with this standard (Berber et al 2019). It is important to note that trials registers are an important source of information about completed, terminated, and ongoing trials, and an increasingly important source of results for completed and terminated trials, especially those whose results have not been published (Fain et al 2018, Zarin et al 2019, Nelson et al 2023, Kaul et al 2024). Two studies, however, found that the data in trials register records were frequently either inaccurate and/or not kept up to date and that trial completion status was commonly inaccurate (Fleminger and Goldacre 2018, Speich et al 2021). It is recommended, therefore, not to limit searches of trials registers and trials register portals by trial status. Similarly, it was found that the phase of the trial was often not recorded correctly (Gresham et al 2022). It is recommended, therefore, also not to limit searches of trials registers and trials register portals by trial phase.

Studies have suggested that trials registers are an important source for identifying additional randomized trials (Baudard et al 2017, Banno et al 2020, Alqaidoom et al 2023) and also for identifying additional or discrepant data for published studies (Chen et al 2022, Paladin and Pranić 2022). Although there are many other trials registers and related resources such as portals, those that are considered to be the most important for searching to identify studies for a systematic review are ClinicalTrials.gov and the WHO International Clinical Trials Registry Platform (ICTRP) portal (Pansieri et al 2017). Research has shown that even though ClinicalTrials.gov is included in the WHO ICTRP Search Portal, not all ClinicalTrials.gov records could be successfully retrieved via searches of the ICTRP Search Portal (Glanville et al 2014, Hausner 2014, Knelangen et al 2018). The extent to which this might still be the case with the new ICTRP interface released in its final version in June 2021 and the changes being made under the ClinicalTrials.gov Modernization programme remains to be ascertained. Therefore, the guidance that it is not sufficient to search the ICTRP alone still stands, pending further research. A recent study reviewed the search interfaces of the EU Clinical Trials Register (EUCTR), ClinicalTrials.gov and the WHO ICTRP and offers further insights into how to search these resources (Cooper et al 2021b), as does another recent article (Hunter et al 2022). Guidance for searching these and other trials registers is provided in the online Technical Supplement.

In addition to Cochrane, other organizations also advocate searching trials registers and/or related portals. These include the Agency for Healthcare Research and Quality (AHRQ) in the US, the European Network for Health Technology Assessment (EUnetHTA), the Institute for Quality and Efficiency in Health Care (IQWiG) in Germany, the Institute of Medicine in the US, and the National Institute of Health and Care Excellence (NICE) in the UK (Institute of Medicine 2011, Agency for Healthcare Research and Quality 2014, National Institute for Health and Care Excellence (NICE) 2014, EUnetHTA JA3WP6B2-2 Authoring Team 2019, Institute for Quality and Efficiency in Health Care 2023).

There has been an increasing recognition by funders, governments, institutions and investigators of the importance of registering trials at inception and providing access to trial results. Despite perceptions and even assertions to the contrary, however, there is no global, universal, legal requirement to register clinical trials at inception or at any other stage in the process, and no global, universal, legal requirement to publish trial results. The situation is, however, improving and some countries and organizations have introduced such legislation (Viergever and Li 2015) together with plans for sanctions for non-compliance (Mirjam Keestra et al 2022, Grabenstein 2023).

Efforts have been made by a number of organizations, including organizations representing the pharmaceutical industry and individual pharmaceutical companies, to begin to provide central access to ongoing trials and in some cases trial results on completion, either on a national or international basis. An audit of pharmaceutical companies’ policies on access to trial data, results and methods, however, showed that the commitments made by companies to transparency of trials were highly variable (Goldacre et al 2017). Increasingly, as already noted, trials registers such as ClinicalTrials.gov also contain the results of completed and terminated trials where these have been posted by the investigator(s), not just simply listings of the details of the trial and/or details of associated publications.

MECIR Box 4.3.d Relevant expectations for conduct of intervention reviews

4.3.4 Regulatory agency sources and clinical study reports

For further details on this topic, please refer to Section 1.2.2 of the online Technical Supplement.

A number of organizations, including Cochrane, recommend searching regulatory agency sources for trial information including clinical study reports. Clinical study reports (CSRs) are the reports of clinical trials providing detailed information, submitted in support of marketing authorization applications, on the methods and results or outcomes of clinical trials. The organizations which recommend searching these sources include the Agency for Healthcare Research and Quality (AHRQ) in the US, the Institute for Quality and Efficiency in Health Care (IQWiG) in Germany, and the Institute of Medicine in the US (Institute of Medicine 2011, Agency for Healthcare Research and Quality 2014, Institute for Quality and Efficiency in Health Care 2023). Relevant regulatory agency sources for CSRs or related information include the Australian Therapeutic Goods Administration (TGA), the European Medicines Agency (EMA), Health Canada, the Japanese Pharmaceuticals and Medical Devices Agency (PMDA) and the US Food and Drug Administration (FDA). Details of these are provided in the online Technical Supplement. In addition to providing access to CSRs, regulatory agencies are also a source of trials-register-like records, for example the EMA with respect to the EU Clinical Trials Register, the recently launched European Clinical Trials Information Service/the EU Clinical Trials database, and The European Database on Medical Devices (EUDAMED) (under development); and the FDA with respect to Drugs@FDA and medical devices information from the FDA. These resources are all covered in the online Technical Supplement.

A study (Hodkinson et al 2018, Jefferson et al 2018) that looked at use of regulatory documents in Cochrane Reviews, found that understanding within the Cochrane community was limited and that guidance and support would be required if review authors were to engage with regulatory documents as a source of evidence. Specifically, guidance on how to identify and use data from regulatory sources is needed (Jørgensen et al 2018). For more information about identifying CSRs, see the online Technical Supplement. Further guidance on collecting data from CSRs is provided in Chapter 5, Section 5.5.6.

4.3.5 Other sources

For further details on this topic, please refer to Section 1 of the online Technical Supplement and its subsections.

The term ‘grey literature’ is often used to refer to reports published outside of traditional commercial publishing. In particular, review authors consider searching sources such as dissertations and conference abstracts (see MECIR Box 4.3.e).

Review authors may also consider searching the internet, handsearching journals and searching full texts of journals electronically where available (see online Technical Supplement for details). They should examine previous reviews on the same topic and check reference lists of included studies and relevant systematic reviews (see MECIR Box 4.3.e).

MECIR Box 4.3.e Relevant expectations for conduct of intervention reviews

4.4 Designing search strategies

4.4.1 Introduction to search strategies

This section highlights some of the issues to consider when designing search strategies. Designing search strategies can be complex and the section does not fully address the many complexities in this area. Review teams will benefit from the skills and expertise of a medical/healthcare librarian or information specialist. Many of the issues highlighted relate to both the subject aspects of the search (e.g. the PICO elements) and to the study design (e.g. randomized trials). For a search to be robust, both aspects require attention to be sure that relevant records are not missed.

Issues to consider in planning a search include:

• the nature or type of the intervention(s) being assessed;
• the complexity of the review question and the need to consider additional conceptual frameworks (see Chapter 3 and Chapter 17);
• the time period when any evaluations of the interventions may have taken place (as specified in the review protocol) (see Section 4.4.5);
• any geographic considerations, such as the need to search the African Index Medicus for studies relating to African populations or the Chinese literature for studies in Chinese herbal medicine (see online Technical Supplement);
• whether the review is limited to randomized trials or other study designs are eligible (see Chapter 24);
• whether a validated methodological search filter (for specific study designs) is available (see Section 4.4.7);
• whether unpublished data are to be sought specifically, see Sections 4.3.2, 4.3.3 and 4.3.4; and
• whether the review has specific eligibility criteria around study design to address adverse effects (see Chapter 19), economic evidence (see Chapter 20) or qualitative evidence (see Chapter 21), in which case searches to address these criteria should be undertaken (see MECIR Box 4.4.a).

Further evidence-based information about designing search strategies can be found on the SuRe Info portal, which is updated twice per year (Isojarvi and Glanville 2021).

MECIR Box 4.4.a Relevant expectations for conduct of intervention reviews

4.4.2 Structure of a search strategy

The starting point for developing a search strategy is to consider the main concepts being examined in a review. This is often referred to as PICO – that is Patient (or Participant or Population or Problem), Intervention, Comparison and Outcomes (Richardson et al 1995, Cullen et al 2023, Hateley 2023): see also Chapter 2 and Chapter 3 for guidance on developing and refining PICO definitions that will be operationalized in the search strategy. Examples are provided in the appendices to the Cochrane Information Specialists’ Handbook (Cochrane Information Specialist Support Team 2021c). For a Cochrane Review of interventions, the review objective should provide the PICO concepts, and the eligibility criteria for studies to be included will further assist in the selection of appropriate subject headings and text words for the search strategy.

The structure of search strategies in bibliographic databases should be informed by the main concepts of the review (see Chapter 3), using appropriate elements from PICO and the study design (see MECIR Box 4.4.b). It is usually unnecessary, however, and may even be undesirable, to search on every aspect of the review’s clinical question (Frandsen et al 2020). Although a research question may specify particular comparators or outcomes, these concepts may not be well described in the title or abstract of an article (or indeed may not be mentioned at all) and are often not well indexed with controlled vocabulary terms such as Medical Subject Headings (MeSH) (Tsujimoto et al 2022)). Therefore, when searching general databases, such as MEDLINE, a search strategy will typically have three sets of terms: (i) terms to search for the health condition of interest, i.e. the population; (ii) terms to search for the intervention(s) evaluated; and (iii) terms to search for the types of study design to be included. Typically, a broad set of search terms will be gathered for each concept and combined with the OR Boolean operator to achieve sensitivity within concepts. The results for each concept are then combined using the AND Boolean operator, to ensure each concept is represented in the final search results.

It is important to consider the structure of the search strategy on a question-by-question basis. In some cases it may be possible and reasonable to search for the comparator, for example if the comparator is explicitly placebo and reported consistently. In other cases, investigation might show that the outcomes are particularly well defined and consistently reported in abstracts, which might justify the inclusion of the outcomes concept in the search. When searchers are considering including outcomes or comparators in the strategy only because of the volume of records being returned (Frandsen et al 2022), it is worth consulting the range of alternative approaches listed as bullet points below. The advice on whether or not to search for outcomes for adverse effects differs from the advice given above (see Chapter 19).

MECIR Box 4.4.b Relevant expectations for conduct of intervention reviews

For some reviews it may not be easy to use the PICO concepts to structure the search in the way described above. This may particularly be the case for reviews addressing complex or unknown interventions or public health interventions (see also Chapter 17 on intervention complexity). Furthermore, the PICO framework is generally not suitable for questions around diagnostic test accuracy, prognosis, qualitative data or methods (Irvin and Hayden 2006, Booth 2016b, Spijker et al 2023, Hirt et al 2024b) (see also Chapter 21 on qualitative evidence), and reviews taking specific approaches, such as realist reviews, may require iterative searches and multiple search strategies (Booth et al 2020a). The term ‘tailored approach’ has been suggested for searches for complex topics or review methods that may not fit into a PICO structure (Cooper et al 2022b). Some options to explore for these situations include:

• consider whether a non-PICO question breakdown may be helpful from which concepts can be selected and tested. A list of question structures and their component concepts is available at: https://gh.bmj.com/content/bmjgh/4/Suppl_1/e001107/DC1/embed/inline-supplementary-material-1.pdf  (Booth et al 2019); 
• use a single concept such as searching for the intervention alone (European Food Safety Authority 2010);
• break a compound concept into two or more subconcepts, for example computer-assisted instruction might be broken down into the concept of computer-assisted and the concept of instruction; 
• use a multi-stranded or multi-faceted approach that uses a series of searches, with different combinations of concepts, to capture a complex research question (Lefebvre et al 2013);
• use a variety of different search approaches to compensate for when a specific concept is difficult to define (Shemilt et al 2014);
• use iterative searches (Bravata et al 2005, Zwakman et al 2018, Booth et al 2020b);
• use citation searching on key articles in addition to a database search (Haddaway et al 2015, Hinde and Spackman 2015) (see online Technical Supplement);
• consider machine learning options whereby broad, sensitive searches are conducted and the search results are categorized or prioritized using machine learning software (Muller et al 2021, Shemilt et al 2021, Cawley 2022, Shemilt et al 2022, Stansfield et al 2022) (see online Technical Supplement);
• consider using visual search strategy building aids such as 2DSearch (https://www.2dsearch.com/), searchrefiner (https://ielab.io/searchrefiner/)  or Nested Knowledge (https://nested-knowledge.com/) that may facilitate exploring the impact of searching using different combinations of concepts (Scells and Zuccon 2018, Svarre and Russell-Rose 2022); and
• consider using AI chatbots such as ChatGPT (https://openai.com/chatgpt/) or Claude (https://claude.ai) to ask for suggestions for ways to structure searches, using a detailed explanation of the search question as a prompt (Alaniz et al 2023, Atlas 2023, Enomoto et al 2023, Wang et al 2023, Zhang 2023, Guimarães et al 2024) (see online Technical Supplement).

In the process of updating reviews, the PICO elements and any impact on search structure should also be reviewed and updated if necessary (Bendersky et al 2022).

4.4.3 Sensitivity versus precision

Searches for systematic reviews aim to be as extensive as possible in order to ensure that as many of the relevant studies as possible are included in the review. It is, however, necessary to strike a balance between striving for comprehensiveness and maintaining relevance when developing a search strategy.

The properties of searches are often quantified using ‘sensitivity’ (also called ‘recall’) and ‘precision’ (see Table 4.4.a). Sensitivity is defined as the number of relevant reports identified divided by the total number of relevant reports in the resource. Precision is defined as the number of relevant reports identified divided by the total number of reports identified. Increasing the comprehensiveness (or sensitivity) of a search will reduce its precision and will usually retrieve more non-relevant reports.

Searches for Cochrane Reviews should seek to maximize sensitivity whilst striving for reasonable precision (see MECIR Box 4.4.b). Article abstracts identified through a database search can usually be screened very quickly to ascertain potential relevance. At a conservatively estimated reading rate of one or two abstracts per minute, the results of a database search can be screened at the rate of 60–120 per hour (or approximately 500–1000 over an 8-hour period), so the high yield and low precision associated with systematic review searching may not be as daunting as it might at first appear in comparison with the total time to be invested in the review.

Table 4.4.a Sensitivity and precision of a search

4.4.4 Controlled vocabulary and text words

For further details on this topic, please refer to  Section 3.2 of the online Technical Supplement and its subsections.

MEDLINE and Embase (and many other databases) can be searched using a combination of two retrieval approaches. One is based on text words, that is terms occurring in the title, abstract or other relevant fields available in the database. The other is based on standardized subject terms assigned to the references either by indexers (specialists who appraise the articles and describe their topics by assigning terms from a specific thesaurus or controlled vocabulary) or automatically using automated indexing approaches. Searches for Cochrane Reviews should use an appropriate combination of these two approaches, i.e. text words and controlled vocabulary (see MECIR Box 4.4.c). Approaches for identifying text words and controlled vocabulary to combine appropriately within a search strategy, including text mining approaches, are presented in the online Technical Supplement.

MECIR Box 4.4.c Relevant expectations for conduct of intervention reviews

4.4.5 Language, date and document format restrictions

Searches should capture as many studies as possible that meet the eligibility criteria, ensuring that relevant time periods and sources are covered and not restricted by language or publication status (see MECIR Box 4.3.a). Review authors should justify the use of any restrictions in the search strategy on publication date and publication format (see MECIR Box 4.4.d).

To reduce the risk of introducing bias, searches should not be restricted by language. By including all languages, conservative search strategies avoid the risk of excluding study records with missing, inconsistent, or incorrect values in their metadata (Aali and Shokraneh 2021). It has also been argued that, when language restrictions are justified, these should not be imposed by limiting the search but by including language as an eligibility criterion during study selection (Pieper and Puljak 2021). Recommendations for rapid reviews searches to limit publication language to English and add other languages only when justified (Garritty et al 2021) are supported by evidence that excluding studies published in languages other than English does not change the conclusions of most systematic reviews (Morrison et al 2012, Jiao et al 2013, Hartling et al 2017, Nussbaumer-Streit et al 2020, Dobrescu et al 2021). However, exceptions that studies published in languages other than English do influence review findings have been observed for complementary and alternative medicine (Moher et al 2003, Pham et al 2005, Wu et al 2013), psychiatry, rheumatology and orthopaedics (Egger et al 2003).

Studies have identified a risk of introducing bias by including lower quality trials published in languages other than English in systematic reviews (Jüni et al 2002, Egger et al 2003), but similar evaluations found only minor quality differences between reports of trials published in English and those published in languages other than English (Moher et al 2003). Additionally, when searches are limited to English or to databases containing only English-language articles, there is a risk that eligible studies may be missed from countries where a particular intervention of interest is more common (e.g. traditional Chinese medicines) (Pilkington et al 2005, Morrison et al 2012). Searches restricted to English-language databases or trials registers may also fail to retrieve all eligible study records of drug interventions (indexing bias) or those reporting negative results (Jia et al 2020). For further discussion of these issues see Chapter 13.

Particularly when resources and time are available, the inclusion of studies published in languages other than English in systematic reviews is recommended to minimize the risk of language or indexing bias (Egger et al 1997, Pilkington et al 2005, Morrison et al 2012, Jia et al 2020). Consequently, Cochrane author teams should plan at the protocol stage not to restrict the search by language (see MECIR Box 4.3.a) and to consider searching databases and trials registers published in languages other than English. If warranted, author teams should engage a medical/healthcare librarian or information specialist who is capable of designing and executing search strategies in these sources.

If a Cochrane Review team requires help with translation of or data extraction from reports of studies published in languages other than English, they should seek assistance to do so (this is a common task for which volunteer assistance can be sought via Cochrane Engage (previously known as Cochrane’s TaskExchange platform), accessible to both Cochrane and non-Cochrane review teams. Where it is not possible to extract the relevant information and data from reports published in languages other than English, readers should be informed of the existence of other possibly relevant reports by adding such reports to ‘studies awaiting classification’ rather than ‘excluded studies’. This information should be reflected in the PRISMA flow diagram (or, if there is no flow diagram, then in the text of the review) as ‘studies awaiting classification’.

Date limits may be used to focus searches (Cooper et al 2018c) as long as the restriction is reported and justified (Egger et al 2003) (see MECIR Box 4.4.d). Further use of a supportive narrative may help explain why a particular date restriction was applied (Craven and Levay 2011, Cooper et al 2018a). For example, a database date restriction of 1989-current for a review of nurse-led community training of epinephrine autoinjectors is justified because this is the approval date of the first device (Center for Drug Evaluation and Research 1989). A date limit may be safely applied in this case as any references published before this date would not meet the review’s selection criteria.

While evidence supports that arbitrary date restrictions (e.g. last 20 years) have little impact on the results for rapid reviews of diagnostic test accuracy studies (Furuya-Kanamori et al 2023), other research demonstrated comparative effect size changes due to date limits in systematic reviews (Marshall et al 2018b). Given a potential loss of data from eligible studies, the application of arbitrary date limits is not recommended for Cochrane systematic reviews of interventions.

Caution should be exercised when designing database search strategies with date restrictions. Information specialists should be aware of the various date fields available from database providers (e.g. create date, entry date, last update date, publication date) as well as the coverage dates of the datafiles searched. It may be necessary to search additional sources or datafiles to ensure adequate coverage of the date period of interest for the review. To account for inconsistent publication dates in database records (e.g. a record for an electronic version of a publication may have an earlier publication date than the print version), search strategies should be restricted to a wider date range than the period of interest for the review.

As any information about an eligible study may contain valuable details for analysis, document format restrictions should not be applied to systematic review searches. For example, excluding letters is not recommended because letters may contain important additional information relating to an earlier trial report or new information about a trial not reported elsewhere (Iansavichene et al 2008). In addition, articles indexed as ‘Comments’ should not be routinely excluded without further examination as these may contain early warnings of possible future retraction (see Section 4.4.6).

As with comments and letters, preprints (versions of scientific articles that precede formal peer review and publication in a journal) should also be considered a potentially relevant source of study evidence, particularly for emerging topics where little evidence exists. Recent and widespread availability of preprints has resulted from an urgent demand for emerging evidence during the COVID-19 pandemic (Gianola et al 2020, Kirkham et al 2020, Callaway 2021, Fraser et al 2021). Recent studies (Zeraatkar et al 2022, Davidson et al 2023) indicated that there was no compelling evidence that preprints provide results that are inconsistent with published articles. As study data are often reported in multiple publications and may be reported differently in each (Oikonomidi et al 2020), efforts to identify all reports for eligible studies, regardless of publication format, are necessary to support subsequent stages of the review process to select, assess and analyse complete study data.

However, practical problems have been observed with regard to including preprints in systematic or rapid reviews. These problems include limited search features or download capabilities of preprint archives (Hoy 2020, Brietzke et al 2023), difficulties reporting reproducible strategies (Brietzke et al 2023), and challenges incorporating different versions of preprints and their final publications (Clyne et al 2021). Comparison studies of preprints and their peer-reviewed publications have also revealed differences in authorship, reported funding and conflicts of interests (Itani et al 2022). As pertinent study information may be reported in peer-reviewed publications and missing from the preprint references, searches should be conducted prior to manuscript completion for peer-reviewed references of any included preprint studies.

Similarly, search limits should not be applied to exclude articles published in presumed ‘predatory journals’. The direction of bias associated with non-inclusion of studies published in predatory journals depends on whether they are publishing valid studies with null results or studies whose results are biased towards finding an effect (see Chapter 7) (Boulos et al 2022).

Further evidence-based information about language, date and other limits can be found on the SuRe Info portal, which is updated twice per year (Isojarvi and Glanville 2021). Assistance with designing strategies in preprint servers is available via this updated syntax table from the University of Victoria, Canada.

MECIR Box 4.4.d Relevant expectations for conduct of intervention reviews

4.4.6 Identifying fraudulent studies, other retracted publications, errata and comments

For further details on this topic, please refer to Section 3.9 of the online Technical Supplement.

When considering the eligibility of studies for inclusion in a Cochrane Review, it is important to be aware that publications for some studies may have been found to contain errors or to be fraudulent or may, for other reasons, have been corrected, retracted or had expressions of concern associated with them since publication. Review authors should examine any relevant retraction notices and errata for information (MECIR Box 4.4.e). This applies both to ‘new’ studies identified for inclusion in a review and to studies that are already included in a review when the review is updated. For review updates, it is important to search MEDLINE and Embase for the latest version of the citations to the records for the (previously) included studies, in case the publications have since been corrected or retracted.

Errata are published to correct unintended errors (accepted as errors by the author(s)) that do not invalidate the conclusions of the article.  Retractions are defined by the Committee on Publication Ethics (COPE) Council’s retraction guidelines (Committee on Publication Ethics (COPE) Council 2019) as “… a mechanism for correcting the literature and alerting readers to articles that contain such seriously flawed or erroneous content or data that their findings and conclusions cannot be relied upon. Unreliable content or data may result from honest error, naïve mistakes, or research misconduct.” Recent studies have shown that scientific misconduct was the most common reason for retractions, specifically duplication, plagiarism and fabrication of data (Gaudino et al 2021, Kohl and Faggion 2024) and that systematic reviews and guidelines were often not corrected to take into account retractions of publications relating to the included studies (Kataoka et al 2022, Bakker et al 2024). Comments are published under a range of circumstances including when errors are suggested by others and also for early concerns regarding suspected misconduct. The US National Library of Medicine, in its MeSH Scope Note, defines expression of concern as: “A notification about the integrity of a published article that is typically written by an editor and should be labelled prominently in the item title.”

Including data from studies that are fraudulent or studies that include errors can have an impact on the overall estimates in systematic reviews. There is an increasing awareness of the importance of not including in systematic reviews studies for which one or more publications has/have been retracted, or those with significant errata and how best to avoid this  (Royle and Waugh 2004, Wright and McDaid 2011, Decullier et al 2014, Fanelli et al 2022, Bakker et al 2023, Khademizadeh et al 2023, Bakker et al 2024). Studies have also shown that retractions can decrease the inclusion of such studies in systematic reviews (Mott et al 2019). The retracted status of articles on journal websites and in bibliographic databases, however, is not always consistently displayed (Suelzer et al 2021).  A recent study, however, showed that even when review authors suspect research misconduct, including data falsification, in the trials that they are considering including in their systematic reviews, they do not always report it (Elia et al 2016). Details of how to identify fraudulent studies, other retracted publications, errata, comments, and expressions of concern are described in the online Technical Supplement. Cochrane holds a policy for managing potentially problematic studies and provides associated implementation guidance.

MECIR Box 4.4.e Relevant expectations for conduct of intervention reviews

4.4.7 Search filters

For further details on this topic, please refer to Section 3.6 of the online Technical Supplement and its subsections.

Search filters are search strategies that are designed to retrieve specific types of records, such as those of a particular methodological design. When searching for randomized trials in humans, a validated filter should be used to identify studies with the appropriate design (see MECIR Box 4.4.f). Filters to identify randomized trials for CENTRAL have been developed specifically for databases such as MEDLINE, Embase and CINAHL Plus. CENTRAL, however, aims to contain only reports with study designs possibly relevant for inclusion in Cochrane Reviews, so searches of CENTRAL should not use a trials ‘filter’ or be limited to human studies. The Cochrane MEDLINE and Embase highly sensitive filters are provided here for ease of access; there are also launch links for them on the ISSG Search Filter Resource website (https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/home/rcts). The online Technical Supplement has other filters including the balanced sensitivity and precision filters for MEDLINE.

Box 4.4.a Cochrane Highly Sensitive Search Strategy for identifying randomized trials in MEDLINE: sensitivity-maximizing version (2008 revision); PubMed format

#1          randomized controlled trial [pt] #2          controlled clinical trial [pt] #3          randomized [tiab] #4          placebo [tiab] #5          drug therapy [sh] #6          randomly [tiab] #7          trial [tiab] #8          groups [tiab] #9          #1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8 #10       animals [mh] NOT humans [mh] #11       #9 NOT #10

PubMed search syntax (for Box 4.4.a above):

Box 4.4.b Cochrane Highly Sensitive Search Strategy for identifying randomized trials in MEDLINE: sensitivity-maximizing version (2023 revision); Ovid format

1            exp randomized controlled trial/ 2            controlled clinical trial.pt. 3            randomized.ab. 4            placebo.ab. 5            drug therapy.fs. 6            randomly.ab. 7            trial.ab. 8            groups.ab. 9            1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 10          exp animals/ not humans.sh. 11          9 not 10

Ovid search syntax (for Box 4.4.b above):

Box 4.4.c Cochrane Highly Sensitive Search Strategy for identifying randomized trials in Embase (2023 revision); Embase.com format (adapted from Glanville et al (2019))

1            ‘randomized controlled trial’/exp 2            ‘controlled clinical trial’/de 3            random*:ti,ab,tt 4            ‘randomization’/de 5            ‘intermethod comparison’/de 6            placebo:ti,ab,tt 7            (compare:ti,tt OR compared:ti,tt OR comparison:ti,tt) 8            ((evaluated:ab OR evaluate:ab OR evaluating:ab OR assessed:ab OR assess:ab) AND (compare:ab OR compared:ab OR comparing:ab OR          comparison:ab)) 9            (open NEXT/1 label):ti,ab,tt 10          ((double OR single OR doubly OR singly) NEXT/1 (blind OR blinded OR blindly)):ti,ab,tt 11          ‘double blind procedure’/de 12          (parallel NEXT/1 group*):ti,ab,tt 13          (crossover:ti,ab,tt OR ‘cross over’:ti,ab,tt) 14          ((assign* OR match OR matched OR allocation) NEAR/6 (alternate OR group OR groups OR intervention OR interventions OR patient OR patients OR subject OR subjects OR participant OR participants)):ti,ab,tt 15          (assigned:ti,ab,tt OR allocated:ti,ab,tt) 16          (controlled NEAR/8 (study OR design OR trial)):ti,ab,tt 17          (volunteer:ti,ab,tt OR volunteers:ti,ab,tt) 18          ‘human experiment’/de 19          trial:ti,tt 20          #1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8 OR #9 OR #10 OR #11 OR #12 OR #13 OR #14 OR #15 OR #16 OR #17 OR #18 OR #19 21          (((random* NEXT/1 sampl* NEAR/8 (‘cross section*’ OR questionnaire* OR survey OR surveys OR database or databases)):ti,ab,tt) NOT (‘comparative study’/de OR ‘controlled study’/de OR ‘randomised controlled’:ti,ab,tt OR ‘randomized controlled’:ti,ab,tt OR ‘randomly assigned’:ti,ab,tt)) 22          (‘cross-sectional study’/de NOT (‘randomized controlled trial’/exp OR ‘controlled clinical study’/de OR ‘controlled study’/de OR ‘randomised controlled’:ti,ab,tt OR ‘randomized controlled’:ti,ab,tt OR ‘control group’:ti,ab,tt OR ‘control groups’:ti,ab,tt)) 23          (‘case control*’:ti,ab,tt AND random*:ti,ab,tt NOT (‘randomised controlled’:ti,ab,tt OR ‘randomized controlled’:ti,ab,tt)) 24          (‘systematic review’:ti,tt NOT (trial:ti,tt OR study:ti,tt)) 25          (nonrandom*:ti,ab,tt NOT random*:ti,ab,tt) 26          ‘random field*’:ti,ab,tt 27          (‘random cluster’ NEAR/4 sampl*):ti,ab,tt 28          (review:ab AND review:it) NOT trial:ti,tt 29          (‘we searched’:ab AND (review:ti,tt OR review:it)) 30          ‘update review’:ab 31          (databases NEAR/5 searched):ab 32          ((rat:ti,tt OR rats:ti,tt OR mouse:ti,tt OR mice:ti,tt OR swine:ti,tt OR porcine:ti,tt OR murine:ti,tt OR sheep:ti,tt OR lambs:ti,tt OR pigs:ti,tt OR piglets:ti,tt OR rabbit:ti,tt OR rabbits:ti,tt OR cat:ti,tt OR cats:ti,tt OR dog:ti,tt OR dogs:ti,tt OR cattle:ti,tt OR bovine:ti,tt OR monkey:ti,tt OR monkeys:ti,tt OR trout:ti,tt OR marmoset*:ti,tt) AND ‘animal experiment’/de) 33          (‘animal experiment’/de NOT (‘human experiment’/de OR ‘human’/de)) 34          #21 OR #22 OR #23 OR #24 OR #25 OR #26 OR #27 OR #28 OR #29 OR #30 OR #31 OR #32 OR #33 35          #20 NOT #34

Embase.com search syntax (for Box 4.4.c above):

Box 4.4.d Cochrane Highly Sensitive Search Strategy for identifying randomized trials in Embase (2023 revision); Ovid format (adapted from Glanville et al (2019))

1            exp randomized controlled trial/ 2            controlled clinical trial/ 3            random$.ti,ab. 4            randomization/ 5            intermethod comparison/ 6            placebo.ti,ab. 7            (compare OR compared OR comparison).ti,ab. 8            ((evaluated OR evaluate OR evaluating OR assessed OR assess) AND (compare OR compared OR comparing OR comparison)).ab 9            (open adj label).ti,ab. 10          ((double OR single OR doubly OR singly) adj (blind OR blinded OR blindly)).ti,ab. 11          double blind procedure/ 12          parallel group$1.ti,ab. 13          (crossover OR cross over).ti,ab. 14          ((assign$ OR match OR matched OR allocation) adj5 (alternate OR group$1 OR intervention$1 OR patient$1 OR subject$1 OR participant$1)).ti,ab. 15          (assigned OR allocated).ti,ab. 16          (controlled adj7 (study OR design OR trial)).ti,ab. 17          (volunteer OR volunteers).ti,ab. 18          human experiment/ 19          trial.ti. 20          or/1-19 21          (random$ adj sampl$ adj7 ("cross section$" OR questionnaire$1 OR survey$ OR database$1)).ti,ab. NOT (comparative study/ OR controlled study/ OR randomi?ed controlled.ti,ab. OR randomly assigned.ti,ab.) 22         cross-sectional study/ NOT (exp randomized controlled trial/ OR controlled clinical trial/ OR controlled study/ OR randomi?ed controlled.ti,ab. OR control group$1.ti,ab.) 23         ((case adj control$) AND random$.ti,ab.) NOT randomi?ed controlled.ti,ab. 24         (systematic review.ti,ab. NOT (trial OR study).ti) 25         (non random$ NOT random$).ti,ab. 26         "random field$".ti,ab. 27         (random cluster adj3 sample$).ti,ab. 28         (review.ab. AND review.pt.) NOT trial.ti. 29         ("we searched".ab. AND (review.ti. OR review.pt.)) 30         "update review".ab. 31          (databases adj4 searched).ab. 32          (rat OR rats OR mouse OR mice OR swine OR porcine OR murine OR sheep OR lambs OR pigs OR piglets OR rabbit OR rabbits OR cat OR cats OR dog OR dogs OR cattle OR bovine OR monkey OR monkeys OR trout OR marmoset$1).ti. AND animal experiment/ 33          animal experiment/ NOT (human experiment/ OR human/) 34          or/21-33 35          20 NOT 34

Ovid search syntax (for Box 4.4.d above):

The InterTASC Information Specialists’ Sub-Group Search Filter Resource offers a collection of search filters, focusing predominantly on methodological search filters and providing critical appraisals of some of these filters. The site includes, amongst others, filters for identifying systematic reviews, randomized and non-randomized studies and qualitative research in a range of databases and across a range of service providers (Glanville et al 2006 ).

Increasingly, machine learning models for selected study designs (classifiers) such as randomized controlled trials are built into systematic review software such as EPPI-Reviewer (https://eppi.ioe.ac.uk/CMS/Portals/35/machine_learning_in_eppi-reviewer_v_7_web_version.pdf) and Covidence (https://www.covidence.org/blog/let-machine-learning-find-the-randomized-controlled-trials-faster), or can be designed within systematic review software such as DistillerSR (https://www.distillersr.com/products/modules (Marshall et al 2023). When using the study design classifiers that are available within systematic review software, it is probably best to ensure that the search strategies used to identify studies from databases do not also contain study design filters. A systematic review of ten varied studies that used supervised machine learning to identify “high-quality clinical articles” reported that machine learning, largely using gold standard sets of records from the ACP Journal Club, achieved a best balance sensitivity of 95% with 86% precision in one study (Abdelkader et al 2021). Other studies had higher sensitivity but lower precision. The authors noted that this is a fast-moving field, although it is hampered by the lack of large reference sets of known relevant RCTs. Details of other publications reporting the development of machine classifiers for identifying cluster randomized trials and “scientifically rigorous” research can be found on the machine learning classifiers page of the InterTASC Information Specialists’ Sub-Group Search Filter Resource.

Search filter performance is discussed further in a detailed report (Lefebvre et al 2017). For further discussion around the design and use of search filters, see the online Technical Supplement. Further evidence-based information about search filters generally and for specific types of studies can be found on the SuRe Info portal, which is updated twice per year (Isojarvi and Glanville 2021).

MECIR Box 4.4.f Relevant expectations for conduct of intervention reviews

4.4.8 Peer review of search strategies

It is strongly recommended that search strategies be peer reviewed before the searches are run. Peer review of search strategies is increasingly recognized as a necessary step in designing and executing high-quality search strategies to identify studies for possible inclusion in systematic reviews (Folb et al 2020, Neilson 2021). As discussed elsewhere (Lefebvre and Duffy 2021), the following organizations and documents advocate peer review of searches: The Agency for Healthcare Research and Quality (AHRQ) in the US, the Campbell Collaboration, the Centre for Reviews and Dissemination in the UK, the European Network for Health Technology Assessment (EUnetHTA), the Institute for Quality and Efficiency in Health Care (IQWiG) in Germany, the Institute of Medicine in the US, the National Institute of Health and Care Excellence (NICE) in the UK, the Preferred Reporting Items for Systematic reviews and Meta-Analyses – Extension for Searches (PRISMA-S Extension) and the PRISMA 2020 explanation and elaboration documents (Centre for Reviews and Dissemination 2009, Institute of Medicine 2011, Agency for Healthcare Research and Quality 2014, National Institute for Health and Care Excellence (NICE) 2014, EUnetHTA JA3WP6B2-2 Authoring Team 2019, The Methods Group of the Campbell Collaboration 2019, Page et al 2021a, Rethlefsen et al 2021, Institute for Quality and Efficiency in Health Care 2023, MacDonald et al 2024). Additionally, the Campbell Collaboration includes an item on peer review of search strategies in its Information Retrieval Methods Group Checklist in its searching guidance (MacDonald et al 2024).

Studies have shown that errors occur in the search strategies underpinning systematic reviews and that search strategies are not always conducted or reported to a high standard (Sampson and McGowan 2006, Mullins et al 2014, Layton 2017, Salvador-Olivan et al 2019, Masterson and Martinez-Silveira 2022, Ramirez et al 2022). This has also been shown to be the case within some Cochrane Reviews (Franco et al 2018, Price et al 2022). The PRISMA-S Extension states that authors “should strongly consider having the search strategy peer reviewed by an experienced searcher, informational specialist, or librarian” (Rethlefsen et al 2021) and encourages authors to consider using the Peer Review of Electronic Search Strategies (PRESS) Guideline Statement and Checklist (McGowan et al 2016b). Research has shown that peer review using a specially designed checklist can improve the quality of searches both in systematic reviews (Relevo and Paynter 2012, Spry et al 2013) and in rapid reviews (Spry et al 2013, Spry and Mierzwinski-Urban 2018). An evidence-based checklist such as the PRESS Evidence-Based Checklist should be used to assess which elements are important in peer review of electronic search strategies (McGowan et al 2016a, McGowan et al 2016b). The PRESS Checklist covers not only the technical accuracy of the strategy (line numbers, spellings, etc.), but also whether the search strategy addresses all relevant aspects of the protocol and has interpreted the research question appropriately. Recently, a Search Submission and Peer Review Assessment form has been made available within the PRESS materials, to improve the peer review process.

It is recommended that authors provide information on the search strategy development and peer review processes. The PRISMA 2020 explanation and elaboration article and the PRISMA-S Extension provide guidance on how and where authors should describe the processes used to develop and validate or peer review the search strategy (Page et al 2021a, Rethlefsen et al 2021). For example, the PRISMA 2020 explanation and elaboration article (Page et al 2021a) states that “the description of the search strategy development process might include details of the approaches used to identify keywords, synonyms, or subject indexing terms used in the search strategies or any processes used to validate or peer review the search strategies”, and “if the search strategy was peer reviewed, report the peer review process used and specify any tool used, such as the Peer Review of Electronic Search Strategies (PRESS) checklist” (McGowan et al 2016b). In the example for Item 7 (Search strategy) of the PRISMA 2020 checklist, the authors propose using the following statement: “The strategy was developed by an information specialist and the final strategies were peer reviewed by an experienced information specialist within our team” (Page et al 2021b). The PRISMA-S Extension (Rethlefsen et al 2021) advocates that the use of peer review be reported and described in the methods section, and proposes the following statement in Item 14 (Peer review: Describe any search peer review process): “The strategies were peer reviewed by another senior information specialist prior to execution using the PRESS Checklist” (McGowan et al 2016b). For Cochrane Reviews, the names, credentials, and institutions of the peer reviewers of the search strategies should be noted in the review (with their permission) in the Acknowledgments section.

Further evidence-based information about peer reviewing search strategies can be found on the SuRe Info portal, which is updated twice per year (Isojarvi and Glanville 2021).

4.4.9 Alerts

Alerts, also called literature surveillance services, ‘push’ services or SDIs (selective dissemination of information), are an excellent method of staying up to date with the medical literature currently being published and with other resources such as trials register resources, as a supplement to designing and running specific searches for specific reviews. In practice, alerts are based on a previously developed search strategy, which is saved in a personal account on the database platform (e.g. ‘My EBSCOhost – search alerts’ on EBSCO, ‘My Searches & Alerts’ on Ovid and ‘MyNCBI – saved searches’ on PubMed). These saved strategies filter the content as the database (or other resource) is being updated with new information. The account owner is notified (usually via email) when new or updated publications or records meeting their specified search parameters are added to the database (or other resource). In the case of PubMed, the alert can be set up to be delivered weekly or monthly, or in real-time and can comprise email or RSS feeds.

For review authors, alerts are a useful tool to help monitor their review topic area after the original search has been conducted. By following an alert, authors can become aware of a new study that might meet the review’s eligibility criteria, and decide either to include it in the review immediately or to mention it in the review’s ‘studies awaiting classification’ section, for inclusion during the next review update (see online Chapter IV). Authors should consider setting up alerts so that the review can be as current as possible at the time of publication.

Another way of attempting to stay current with the literature as it emerges is by using alerts based on journal tables of contents (TOCs). These usually cannot be specifically tailored to the information needs in the same way as search strategies developed to cover a specific topic. They can, however, be a good way of trying to keep up to date on a more general level by monitoring what is currently being published in journals of interest. Many journals, even those that are available by subscription only, offer TOC alert services free of charge. In addition, a number of publishers and organizations offer TOC services (see online Technical Supplement). Use of TOCs is not proposed as a single alternative to the various other methods of study identification necessary for undertaking systematic reviews, rather as a supplementary method (See also Chapter 22, Sections 22.2 and 22.3 for a discussion of new technologies to support evidence surveillance in the context of ‘living’ systematic reviews).

As mentioned above, alerts should also be considered for sources beyond databases and journal TOCs, such as trials register resources and regulatory information.

4.4.10 Timing of searches

The published review should be as up to date as possible. Searches for all the relevant sources (databases, trials registers, etc.) should be rerun prior to publication, if the initial search date is more than 12 months (preferably six months) from the intended publication date (see MECIR Box 4.4.g). The results should also be screened to identify potentially eligible studies. Ideally, the studies should be incorporated fully in the review. If not, then the potentially eligible studies will need to be reported as references under ‘studies awaiting classification’ (or under ‘ongoing studies’ if they are not yet completed).

MECIR Box 4.4.g Relevant expectations for conduct of intervention reviews

4.4.11 When to stop searching

Developing a search within a database is often an iterative and exploratory process. It involves exploring trade-offs between search terms and assessing their overall impact on the sensitivity and precision of the search. It is often difficult to decide in a scientific or objective way when a search is complete and search strategy development can stop. The ability to decide when to stop typically develops through experience of developing many strategies. Suggestions for stopping rules have been made around the retrieval of new records, for example to stop if adding in a series of new terms to a database search strategy yields no new relevant records, or if precision falls below a particular cut-off point (Chilcott et al 2003). Stopping might also be appropriate when the removal of terms or concepts results in missing relevant records. Another consideration is the amount of evidence that has already accrued: in topics where evidence is scarce, authors might need to be more cautious about deciding when to stop searching. Although methods have been described to assist with deciding when to stop developing the search, there has been little formal evaluation of the approaches (Arber et al , Booth 2010).

Research has demonstrated that searches within databases may miss relevant studies and that careful attention should be paid to achieving a sensitive search strategy since there may always be room for improvement (Matthews et al 1999, Savoie et al 2003, Booth 2016a). At a basic level, investigation is needed as to whether a strategy is performing adequately. One simple test is to check whether the search is finding the publications that have been recommended as key publications or that have been included in other similar reviews (Cooper et al 2018b, EUnetHTA JA3WP6B2-2 Authoring Team 2019). It is not enough, however, for the strategy to find only those records, otherwise this might be a sign that the strategy is biased towards known studies and other relevant records might be being missed. In addition, citation searches (see online Technical Supplement Section 1.1.4) and reference checking (see online Technical Supplement Section 1.3.4) are useful checks of strategy performance. If those additional methods are finding documents that the searches have already retrieved, but that the team did not necessarily know about in advance, then this is one sign that the strategy might be performing adequately. Also, an evidence-based checklist such as the PRESS Evidence-Based Checklist (McGowan et al 2016b) should be used to assess whether the search strategy is adequate (see Section 4.4.8). If some of the PRESS dimensions seem to be missing without adequate explanation or arouse concerns, then the search may not yet be complete.

Statistical techniques can be used to assess performance, such as capture-recapture (Spoor et al 1996, Ferrante di Ruffano et al 2012), also known as capture-mark-recapture (Kastner et al 2009, Lane et al 2013), or the relative recall technique (Sampson et al 2006, Sampson and McGowan 2011). Kastner suggests the capture-mark-recapture technique merits further investigation since it could be used to estimate the number of studies in a literature prospectively and to determine where to stop searches once suitable cut-off levels have been identified. Kastner’s approach involves searching databases, conducting record selection, calculating capture-mark-recapture and then making decisions about whether further searches are necessary. This would entail potentially an iterative search and selection process. Capture-recapture needs results from at least two searches to estimate the number of missed studies. Further investigation of published prospective techniques seems warranted to learn more about the potential benefits.

Relative recall (Sampson et al 2006, Sampson and McGowan 2011) requires a range of searches to have been conducted so that the relevant studies have been built up by a set of sensitive searches. The performance of the individual searches can then be assessed in each individual database by determining how many of the studies that were deemed eligible for the evidence synthesis and were indexed within a database, can be found by the database search used to populate the synthesis. If a search in a database did not perform well and missed many studies, then that search strategy is likely to have been suboptimal. If the search strategy found most of the studies that were available to be found in the database, then it was likely to have been a sensitive strategy. Assessments of precision could also be made, but these mostly inform future search approaches since they cannot affect the searches and record assessment already undertaken. Relative recall may be most useful at the end of the search process since it relies on the achievement of several searches to make judgements about the overall performance of strategies.

Citation searches (see online Technical Supplement) are another way to assess whether the search is complete. Citation searches, conducted using the included (eligible) studies identified after a series of searches, may reveal additional relevant studies. If the citation searches yield a large number of additional relevant records, this may suggest that the original searches were not optimal and should be revisited.

In evidence synthesis involving qualitative data, searching is often more organic and intertwined with the analysis such that the searching stops when new information ceases to be identified (Booth 2016b). The reasons for stopping need to be documented and it is suggested that explanations or justifications for stopping may centre around saturation (Booth 2016b). Further information on searches for qualitative evidence can be found in Chapter 21.

In searches for complex topics or in those types of review where there are different approaches to the completeness of study identification, exhaustive structured searches of bibliographic databases may not be the highest priority and so the issue of knowing when to stop may take on a different focus (Cooper et al 2018b, Cooper et al 2022b). A range of approaches to stopping searches specific to living reviews has been suggested (Heron et al 2023).

4.5 Documenting and reporting the search process

Review authors should document the search process in enough detail to ensure that it can be reported correctly in the review (see MECIR Box 4.5.a). The searches of all resources should be reproducible to the extent that this is possible. By documenting the search process, we refer to internal record-keeping, which is distinct from reporting the search process in the review (discussed in online Chapter III).

MECIR Box 4.5.a Relevant expectations for conduct of intervention reviews

Medical/healthcare librarians and information specialists involved with the review should draft, or at least comment on, the search strategy sections of the review prior to publication (Pauwels et al 2024).

Suboptimal reporting of systematic review search activities and methods has been observed (Sampson et al 2008, Roundtree et al 2009, Niederstadt and Droste 2010, Shamsi et al 2020, Rethlefsen et al 2024). In one report, ninety percent of systematic reviews failed to report search methods adequately (de Kock et al 2020). Research has also shown a lack of compliance with guidance in the Handbook with respect to search strategy description in published Cochrane Reviews (Sampson and McGowan 2006, Yoshii et al 2009, Wasiak et al 2017, Franco et al 2018, Nguyen et al 2022, Uttley et al 2023). The lack of consensus regarding optimal reporting has been a challenge with respect to the values of transparency and reproducibility (Rethlefsen et al 2024). The PRISMA-Search (PRISMA-S) Extension (Rethlefsen et al 2021), an extension to the PRISMA Statement (Page et al 2021b, Page et al 2021a), addresses the reporting of search strategies in systematic reviews. PRISMA-S (together with the major revision of PRISMA itself) provides enough detail and specific examples for systematic review authors to report search methods and information sources in a clear, reproducible way. In Box 2 of the PRISMA 2020 guidance under “Noteworthy changes to the PRISMA 2009 statement” the guidance has been strengthened to stipulate: “Modification of the ‘Search’ item to recommend authors present full search strategies for all databases, registers and websites searched, not just at least one database (see item #7)”. This brings the PRISMA 2020 guidance more into line with Cochrane standards for reporting of search strategies. Further PRISMA-related guidance has been published relating to tracking records and completion of the PRISMA flow diagram (Rethlefsen and Page 2022).

There is also a recommendation in the PRISMA 2020 guidance (see item 27) that “authors state whether data used in the review are publicly available and if so, where they can be accessed” (Page et al 2021a). These recommendations may influence record keeping practices of searchers. Additionally, a proposed data structure for transparent and repeatable reporting of bibliographic searching has been published (Haddaway et al 2022).

It is recommended that review authors seek guidance from a medical/healthcare librarian or information specialist at the earliest opportunity with respect to documenting the search process (Rethlefsen et al 2015, Faggion et al 2016, Meert et al 2016, Pauwels et al 2024, Pawliuk et al 2024). For Cochrane Reviews, the bibliographic database search strategies should all be copied and pasted into the search strategies supplementary material exactly as run and in full, together with the search set numbers and the total number of records retrieved by each search strategy. The search strategies should not be re-typed, because this can introduce errors. The same process, to the extent that it is possible, is also good practice for searches of trials registers and other sources, where the interface used, such as introductory or advanced, should also be specified. Creating a report of the search process can be accomplished through methodical documentation of the steps taken by the searcher. This need not be onerous if suitable record keeping is performed during the process of the search, but it can be nearly impossible to recreate post hoc. Many database interfaces have facilities for search strategies to be saved online or to be emailed; an offline copy in text format should also be saved. For some databases, taking and saving a screenshot of the search may be the most practical approach (Rader et al 2014).

Documenting the searching of sources other than databases, including the search terms used, is also required if searches are to be reproducible (Atkinson et al 2015, Chow 2015, Witkowski and Aldhouse 2015, Booth et al 2020b, Hirt et al 2024a). Guidance on terminology, application, and reporting of citation searching has recently been published (the TARCiS statement) (Hirt et al 2024a).

Details about contacting experts or manufacturers, searching reference lists, scanning websites, and decisions about search iterations can be produced as an appendix in the final document and used for future updates. The purpose of search documentation is transparency, internal assessment, and reference for any future update. It is important to plan how to record searching of sources other than databases since some activities (contacting experts, reference list searching, and forward citation searching) will occur later on in the review process after the database results have been screened (Rader et al 2014). The searcher should record any correspondence on key decisions and report a summary of this correspondence alongside the search strategy in a search narrative. The narrative describes the major decisions that shaped the strategy and can give a peer reviewer an insight into the rationale for the search approach (Craven and Levay 2011). A worked example of a search narrative is available (Cooper et al 2018a).

It is particularly important to save locally or file print copies of any information found on the internet, such as information about ongoing and/or unpublished trials, as this information may no longer be accessible at the time the review is written. Local copies should be stored in a structured way to allow retrieval when needed. There are also web-based tools which archive webpage content for future reference, such as WebCite (Eysenbach and Trudel 2005). The results of web searches will not be reproducible to the same extent as bibliographic database searches because web content and search engine algorithms frequently change, and search results can differ between users due to a general move towards localization and personalization (Cooper et al 2021a). It is still important, however, to document the search process to ensure that the methods used can be transparently reported (Briscoe 2018). In cases where a search engine retrieves more results than it is practical to screen in full (it is rarely practical to search thousands of web results, as the precision of web searches is likely to be relatively low), the number of results that are documented and reported should be the number that were screened rather than the total number (Dellavalle et al 2003, Bramer 2016).

Decisions should be documented for all records identified by the search. Details of the flow of studies from the number(s) of references identified in the search to the number of studies included in the review will need to be reported in the final review, ideally using a flow diagram such as that proposed in the PRISMA guidance (see online Chapter III); these can be generated using software including Covidence, DistillerSR, EPPI-Reviewer, the METAGEAR package for R, the PRISMA Flow Diagram Generator, and RevMan. A table of ‘Characteristics of excluded studies’ will also need to be presented (see Section 4.6.5). Numbers of records are sufficient for exclusions based on initial screening of titles and abstracts. Broad categorizations are sufficient for records classed as potentially eligible during an initial screen of the full text. Authors will need to decide for each review when to map records to studies (if multiple records refer to one study). The flow diagram records initially the total number of records retrieved from various sources, then the total number of studies to which these records relate. Review authors need to match the various records to the various studies in order to complete the flow diagram correctly. Lists of included and excluded studies must be based on studies rather than records (see also Section 4.6.1).

Further evidence-based information about documenting and reporting the search process can be found on the SuRe Info portal, which is updated twice per year (Isojarvi and Glanville 2021).

4.6 Selecting studies

4.6.1 Studies (not reports) as the unit of interest

A Cochrane Review is a review of studies that meet pre-specified eligibility criteria. Since each study may have been reported in several articles, abstracts or other reports, an extensive search for studies for the review may identify many reports for each potentially relevant study. Two distinct processes are therefore required to determine which studies can be included in the review. One is to link together multiple reports of the same study; and the other is to use the information available in the various reports to determine which studies are eligible for inclusion. Although sometimes there is a single report for each study, it should never be assumed that this is the case.

As well as the studies that inform the systematic review, other studies will also be identified and these should be recorded or tagged as they are encountered, so that they can be listed in the relevant tables in the review (see Section 4.6.3).

4.6.2 Identifying multiple reports from the same study

Duplicate publication can introduce substantial biases if studies are inadvertently included more than once in a meta-analysis (Tramèr et al 1997). Duplicate publication can take various forms, ranging from identical manuscripts to reports describing different outcomes of the study or results at different time points (von Elm et al 2004). The number of participants may differ in the different publications. It can be difficult to detect duplicate publication and some ‘detective work’ by the review authors may be required.

Some of the most useful criteria for comparing reports are:

• trial identification numbers (e.g. ClinicalTrials.gov Identifier (NCT number); ISRCTN; Universal Trial Number (UTN) (assigned by the ICTRP); other identifiers such as those from the sponsor) (Liu et al 2022, Smalheiser and Holt 2022);
• author names (most duplicate reports have one or more authors in common, although this is not always the case);
• location and setting (particularly if institutions, such as hospitals, are named);
• specific details of the interventions (e.g. dose, frequency);
• numbers of participants and baseline data; and
• date and duration of the study (which can also clarify whether different sample sizes are due to different periods of recruitment).

Where uncertainties remain after considering these and other factors, it may be necessary to correspond with the authors of the reports and/or the principal investigators of the studies for unpublished data.

Multiple reports of the same study should be collated, so that each study, rather than each report, is the unit of interest in the review (see MECIR Box 4.6.a). Review authors will need to choose and justify which report (the primary report) to use as a source for study results, particularly if two reports include conflicting results. They should not discard other (secondary) reports, since they may contain additional outcome measures and valuable information about the design and conduct of the study.

MECIR Box 4.6.a Relevant expectations for conduct of intervention reviews

4.6.3 A typical process for selecting studies

A typical process for selecting studies for inclusion in a review is as follows (the planned process should be detailed in the protocol for the review):

1. Merge search results from different sources using reference management software, and remove duplicate records of the same report (i.e. records reporting the same journal title, volume and page numbers).
2. Examine titles and abstracts to remove obviously irrelevant reports (authors should generally be over-inclusive at this stage).
3. Retrieve the full text of the potentially relevant reports.
4. Link together multiple reports of the same study (see Section 4.6.2).
5. Examine full-text reports for compliance of studies with eligibility criteria.
6. Correspond with investigators, where appropriate, to clarify study eligibility (it may be appropriate to request further information, such as missing methods information or results, at the same time). If study data remain incomplete/unobtainable those studies should be tagged/recorded as incomplete, and should be listed in the table of ‘Characteristics of studies awaiting classification’ in the review.
7. Make final decisions on study inclusion and proceed to data collection.
8. Tag or record (i) any ongoing studies, so that they can be added to the ongoing studies table, and (ii) any completed or terminated studies (including any that are presumed to be completed based on the information available) but not yet reported, so that they can be added to either the included studies table or the table of studies awaiting classification, depending on whether the study clearly meets the review’s eligibility criteria.

Note that studies should not be omitted from a review solely on the basis of measured outcome data not being reported (see MECIR Box 4.6.b and Chapter 13).

MECIR Box 4.6.b Relevant expectations for conduct of intervention reviews

4.6.4 Implementation of the selection process

Decisions about which studies to include in a review are among the most influential decisions that are made in the review process and they involve judgement. Use (at least) two people working independently to determine whether each study meets the eligibility criteria. Ideally, screening of titles and abstracts to remove irrelevant reports should also be done in duplicate by two people working independently (although it is acceptable that this initial screening of titles and abstracts is undertaken by only one person). It is essential, however, that two people working independently are used to make a final determination as to whether each study considered possibly eligible after title/abstract screening meets the eligibility criteria based on the full text of the study report(s) (see MECIR Box 4.6.c).

MECIR Box 4.6.c Relevant expectations for conduct of intervention reviews

It has been shown that using at least two authors may reduce the possibility that relevant reports will be discarded (Edwards et al 2002, Waffenschmidt et al 2019, Gartlehner et al 2020) although other case reports have suggested single screening approaches may be adequate (Doust et al 2005, Shemilt et al 2016). Opportunities for screening efficiencies seem likely to become available through promising developments in single human screening in combination with machine learning approaches (O'Mara-Eves et al 2015).

Experts in a particular area frequently have pre-formed opinions that can bias their assessment of both the relevance and validity of articles (Cooper and Ribble 1989, Oxman and Guyatt 1993). Thus, while it is important that at least one author is knowledgeable in the area under review, it may be an advantage to have a second author who is not a content expert.

Disagreements about whether a study should be included can generally be resolved by discussion. Often the cause of disagreement is a simple oversight on the part of one of the review authors. When the disagreement is due to a difference in interpretation, this may require arbitration by another person. Occasionally, it will not be possible to resolve disagreements about whether to include a study without additional information. In these cases, authors may choose to categorize the study in their review as one that is awaiting assessment until the additional information is obtained from the study authors or investigators.

A single failed eligibility criterion is sufficient for a study to be excluded from a review. In practice, therefore, eligibility criteria for each study should be assessed in order of importance, so that the first ‘no’ response can be used as the primary reason for exclusion of the study, and the remaining criteria need not be assessed. The eligibility criteria order may be different in different reviews and they do not always need to be the same.

For most reviews it will be worthwhile to pilot test the eligibility criteria on a sample of reports (say six to eight articles, including ones that are thought to be definitely eligible, definitely not eligible and doubtful). The pilot test can be used to refine and clarify the eligibility criteria, train the people who will be applying them and ensure that the criteria can be applied consistently by more than one person.

For Cochrane Reviews the selection process must be documented in sufficient detail to be able to complete a flow diagram and a table of ‘Characteristics of excluded studies’ (see MECIR Box 4.6.d). During the selection process it is crucial to keep track of the number of references and subsequently the number of studies so that a flow diagram can be constructed. The decisions and reasons for exclusion can be tracked using reference management software, a simple document or spreadsheet, or using specialist systematic review software (see Section 4.6.6.1).

MECIR Box 4.6.d Relevant expectations for conduct of intervention reviews

4.6.5 Selecting ‘excluded studies’

A Cochrane Review includes a list of excluded studies called ‘Characteristics of excluded studies’, detailing the specific reason for exclusion for any studies that a reader might plausibly expect to see among the included studies. This covers all studies that may, on the surface, appear to meet the eligibility criteria but which, on further inspection, do not. It also covers those that do not meet all of the criteria but are well known and likely to be thought relevant by some readers. By listing such studies as excluded and giving the primary reason for exclusion, the review authors can show that consideration has been given to these studies. The list of excluded studies should be as brief as possible. It should not list all of the reports that were identified by an extensive search. It should not list studies that obviously do not fulfil the eligibility criteria for the review, such as ‘Types of studies’, ‘Types of participants’, and ‘Types of interventions’. In particular, it should not list studies that are obviously not randomized if the review includes only randomized trials. Based on a sample, undertaken in 2017/2018 by one of the authors (JT), of approximately 60% of the intervention reviews in the Cochrane Library which included randomized trials only, the average number of studies listed in the ‘excluded studies’ table was 30.

4.6.6 Software support for selecting studies

An extensive search for eligible studies in a systematic review can often identify thousands of records that need to be manually screened. Selecting studies from within these records can be a particularly time-consuming, laborious and logistically challenging aspect of conducting a systematic review. These and other challenges have led to the development of various software tools (and approaches for using ‘generic’ tools) that offer support for the selection process.

Broadly, software to support selecting studies can be classified as:

• systems that support the study selection process, typically involving multiple reviewers (see Section 4.6.6.1); and
• tools and techniques based on text mining and/or machine learning, which aim to semi- or fully-automate the selection process (see Section 4.6.6.2).

Software to support the selection process, along with other stages of a systematic review, including text mining tools, can be identified using the Systematic Review Toolbox (currently unavailable (September 2024)). The Systematic Review Toolbox is a community driven, web-based catalogue of tools that provide support for systematic reviews (Marshall and Brereton 2015). This service is currently unavailable (September 2024) whilst the developers seek and transition to a new hosting solution.

4.6.6.1 Software for managing the selection process

Managing the selection process can be challenging, particularly in a large-scale systematic review that involves multiple reviewers. Basic productivity tools can help (such as word processors, spreadsheets, and reference management software), and several purpose-built systems that support multiple concurrent users are also available that offer support for the study selection process. Software for managing the selection process can be identified using the Systematic Review Toolbox mentioned above. As noted above, this service is currently unavailable (September 2024) whilst the developers seek and transition to a new hosting solution.

Compatibility with other software tools used in the review process (such as RevMan) may be a consideration when selecting a tool to support study selection. Covidence and EPPI-Reviewer are Cochrane-preferred tools, and are likely to have the strongest integration with RevMan. Should specialist software not be available, Bramer and colleagues have reported a method for using the widely available software EndNote X7 for managing the screening process (Bramer et al 2017).

4.6.6.2 Automating the selection process

Research into automating the study selection process through artificial intelligence (‘AI’), machine learning and text mining has received considerable attention over recent years, resulting in the development of numerous tools and techniques for reviewers to consider. The use of automated tools has the potential to reduce the workload involved with selecting studies significantly (Thomas et al 2017). For example, research suggests that adopting automation can reduce the need for manual screening by at least 30% and possibly more than 90%, although sometimes at the cost of up to a 5% reduction in sensitivity (O'Mara-Eves et al 2015). This section discusses these tools in three main areas: a) those that operate across reviews; b) those that use crowdsourcing to reduce reviewer workload; and c) those that operate within individual reviews.

a) The first class of tool is machine learning models (or ‘classifiers’) that can be built where sufficient data are available. Of particular practical use to Cochrane Review authors is a classifier (the ‘RCT Classifier’) that can identify reports of randomized trials based on titles and abstracts. The classifier is highly accurate because it is built on a large dataset of hundreds of thousands of records screened by Cochrane Crowd, Cochrane’s citizen science platform, where contributors help to identify and describe health research (Marshall et al 2018a, Noel-Storr et al 2021a, Thomas et al 2021). Guidance on using the RCT Classifier in Cochrane Reviews, for example to exclude studies already flagged as not being randomized trials, or to access Cochrane Crowd to assist with screening, is available from the Cochrane Information Specialists’ Handbook (Cochrane Information Specialist Support Team 2021a).

An emerging suite of tools built on ‘Large Language Models’ (LLMs) such as ChatGPT may soon offer practical advantages in automating study selection and other parts of the review process. These models promise the possibility of carrying out ‘zero shot learning’, where records can be classified automatically without the need for any specific training. Such approaches may offer substantial benefits, though, at the time of writing in mid-2023, no sufficiently large and valid evaluations are available.

b) Cochrane has also implemented a screening workflow called Screen4Me (Noel-Storr et al 2021b). This workflow incorporates the use of the RCT Classifier and Cochrane Crowd, to identify the RCTs found in authors’ search results. Cochrane author teams conducting intervention reviews that incorporate RCTs can access this workflow via the Cochrane Register of Studies. Author teams wishing to use the Screen4Me workflow should liaise directly with their Cochrane Group. To date (July 2024), Screen4Me has been used in more than 250 Cochrane intervention reviews. Workload reduction in terms of screening burden varies depending on the prevalence of RCTs in the domain area and the sensitivity of the searches conducted. A recent internal, as yet (July 2024) unpublished, evaluation by one of the authors (AN-S) showed a mean reduction in screening workload of 53% (range 26% to 84%). More information on Screen4Me can be found in the Cochrane Information Specialists’ Handbook (Cochrane Information Specialist Support Team 2021a).

c) In addition to learning from large datasets such as those generated by Cochrane Crowd, it is also possible for machine learning models to learn how to apply eligibility criteria within individual reviews. This approach uses a process called ‘active learning’ and it is able to semi-automate study selection by continuously promoting records most likely to be relevant to the top of the results list (O'Mara-Eves et al 2015). It is difficult for authors to determine in advance when it is safe to stop screening and allow some records to be eliminated automatically without manual assessment. Recent work has suggested that this barrier is not insurmountable, and that it is possible to estimate how many relevant records remain to be found based on the sample already screened (Sneyd and Stevenson 2019, Callaghan and Muller-Hansen 2020, Li and Kanoulas 2020). The automatic elimination of records using this approach has not been recommended for use in Cochrane Reviews at the time of writing in mid-2023, since more work is needed to develop and validate safe ‘stopping rules’. This active learning process can still be useful, however, since by prioritizing records for screening in order of relevance, it enables authors to identify the studies that are most likely to be included much earlier in the screening process than would otherwise be possible.

Recent developments in this class of tools have seen increased support for keeping living systematic reviews (see also Chapter 22) up to date. Work is still ongoing, but the COVID-19 pandemic lent an urgency to tool development in order to maintain surveillance on the rapidly evolving evidence base (Thomas 2021). Evidence surveillance in living systematic reviews may be a fruitful use case for automation, because of the availability of data on which the machine can ‘learn’. Two case studies found that automation to support living reviews in COVID-19 were both accurate and saved manual effort (Shemilt et al 2021, Marshall et al 2023).

Finally, tools are available that use natural language processing to highlight sentences and key phrases automatically (e.g. PICO elements, trial characteristics, details of randomization) to support the reviewer whilst screening (Tsafnat et al 2014).

4.7 Chapter information

Authors: Carol Lefebvre, Julie Glanville, Simon Briscoe, Robin Featherstone, Anne Littlewood, Maria-Inti Metzendorf, Anna Noel-Storr, Robin Paynter, Tamara Rader, James Thomas, L. Susan Wieland; on behalf of the Cochrane Information Retrieval Methods Group

Acknowledgements: This chapter has been developed from sections of previous editions of this Handbook since 1995. Authors of previous editions included Kay Dickersin, Julie Glanville, Kristen Larson, Carol Lefebvre, Eric Manheimer, Chris Marshall and Farhad Shokraneh. Many of the sources listed and issues discussed in this chapter and the accompanying online Technical Supplement have been brought to our attention by a variety of people over the years and we should like to acknowledge this. Additionally, we should like to acknowledge: Ruth Foxlee, (formerly) Information Specialist, Cochrane Editorial Unit; Miranda Cumpston, (formerly) Head of Learning & Support, Cochrane Central Executive; Colleen Finley, Product Manager, John Wiley and Sons, for checking sections relating to searching the Cochrane Library; the (UK) National Institute for Health and Care Excellence, the German Institute for Quality and Efficiency in Health Care (IQWiG) and the authors of a living systematic review on the problems with systematic reviews (Uttley et al 2023) for support in identifying some of the references; the (US) Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare Program Scientific Resource Center for their previous Article Alert service; Tianjing Li, Co-Convenor, Comparing Multiple Interventions Methods Group, for text and references that formed the basis of the re-drafting of parts of Section 4.6 Selecting studies; Lesley Gillespie, Cochrane author and former Editor and Trials Search Co-ordinator of the Cochrane Bone, Joint and Muscle Trauma Group, for copy-editing an early draft; the Cochrane Information Specialists’ Executive, the previous Cochrane Information Specialists’ Support Team, Cochrane Information Specialists, members of the Cochrane Information Retrieval Methods Group and others for comments on drafts; Su Golder, Co-Convenor, Adverse Effects Methods Group and Steve McDonald, Co-Director, Cochrane Australia for peer review of earlier versions.

4.8 References

Aali G, Shokraneh F. No limitations to language, date, publication type, and publication status in search step of systematic reviews. J Clin Epidemiol 2021; 133: 165-167.

Aamodt M, Huurdeman H, Strømme H. Librarian co-authored systematic reviews are associated with lower risk of bias compared to systematic reviews with acknowledgement of librarians or no participation by librarians. Evidence Based Library and Information Practice 2019; 14: 103-127.

Abdelkader W, Navarro T, Parrish R, Cotoi C, Germini F, Iorio A, Haynes RB, Lokker C. Machine learning approaches to retrieve high-quality, clinically relevant evidence from the biomedical literature: systematic review. JMIR Medical Informatics 2021; 9: e30401.

Agency for Healthcare Research and Quality. Methods guide for effectiveness and comparative effectiveness reviews: AHRQ publication no. 10(14)-EHC063-EF. 2014.  https://effectivehealthcare.ahrq.gov/products/collections/cer-methods-guide.

Alaniz L, Vu C, Pfaff MJ. The Utility of Artificial Intelligence for Systematic Reviews and Boolean Query Formulation and Translation. Plast Reconstr Surg Glob Open 2023; 11: e5339.

Alqaidoom Z, Nguyen PY, Awadh M, Page MJ. Impact of searching clinical trials registers in systematic reviews of pharmaceutical and non-pharmaceutical interventions: Reanalysis of meta-analyses. Research Synthesis Methods 2023; 14: 52-67.

Arber M, Wood H, Miller P. Search strategy development. Last updated 12 March 2024. In: SuRe Info: Summarized Research in Information Retrieval for HTA Updated 18 April 2024 [Internet]. Available from: https://www.sure-info.org//search-strategy-development.

Arber M, Cikalo M, Glanville J, Lefebvre C, Varley D, Wood H. Annotated bibliography of published studies addressing searching for unpublished studies and obtaining access to unpublished data. 2013. https://methods.cochrane.org/sites/methods.cochrane.org.irmg/files/public/uploads/Annotatedbibliographtifyingunpublishedstudies.pdf.

Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Research Synthesis Methods 2015; 6: 87-95.

Atlas S. Chatbot Prompting: A guide for students, educators, and an AI-augmented workforce 2023. Available from: https://www.researchgate.net/publication/367464129_Chatbot_Prompting_A_guide_for_students_educators_and_an_AI-augmented_workforce.

Bai Y, Gao J, Zou D, Li Z. Is MEDLINE alone enough for a meta-analysis? Alimentary Pharmacology and Therapeutics 2007; 26: 125-126; author reply 126.

Bakker C, Boughton S, Faggion CM, Fanelli D, Kaiser K, Schneider J. Reducing the residue of retractions in evidence synthesis: ways to minimise inappropriate citation and use of retracted data. BMJ Evid Based Med 2024; 29: 121-126.

Bakker CJ, Theis-Mahon N, Brown SJ, Zeegers MP. The relationship between methodological quality and the use of retracted publications in evidence syntheses. Syst Rev 2023; 12: 168.

Banno M, Tsujimoto Y, Kataoka Y. Using the Cochrane Central Register of Controlled Trials to identify clinical trial registration is insufficient: a cross-sectional study. BMC Medical Research Methodology 2020; 20: 200.

Baudard M, Yavchitz A, Ravaud P, Perrodeau E, Boutron I. Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ 2017; 356: j448.

Bendersky J, Auladell-Rispau A, Urrutia G, Rojas-Reyes MX. Methods for developing and reporting living evidence synthesis. J Clin Epidemiol 2022; 152: 89-100.

Bennett DA, Jull A. FDA: untapped source of unpublished trials. Lancet 2003; 361: 1402-1403.

Berber S, Tan-Koay AG, Opiyo N, Dwan K, Glanville JM, Lasserson TJ, Willson ML. A cross-sectional audit showed that most Cochrane intervention reviews searched trial registers. J Clin Epidemiol 2019; 113: 86-91.

Boberg-Ans LC, Klefter ON, Rasmussen MLR, van Dijk EHC, Subhi Y. Scientific Literature Database Coverage of Randomized Clinical Trials for Central Serous Chorioretinopathy. J Pers Med 2023; 13.

Booth A. How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. INTERNATIONAL JOURNAL OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 2010; 26: 431-435.

Booth A. Over 85% of included studies in systematic reviews are on MEDLINE. J Clin Epidemiol 2016a; 79: 165-166.

Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Systematic reviews 2016b; 5: 74.

Booth A, Noyes J, Flemming K, Moore G, Tunçalp Ö, Shakibazadeh E. Formulating questions to explore complex interventions within qualitative evidence synthesis. BMJ Glob Health 2019; 4: e001107.

Booth A, Briscoe S, Wright JM. The "realist search": A systematic scoping review of current practice and reporting. Res Synth Methods 2020a; 11: 14-35.

Booth A, Briscoe S, Wright JM. The "realist search": A systematic scoping review of current practice and reporting. Research Synthesis Methods 2020b; 11: 14-35.

Boulos L, Rothfus M, Goudreau A, Manley A. A descriptive study found low prevalence of presumed predatory publications in a subset of Cochrane reviews. J Clin Epidemiol 2022; 152: 316-325.

Bramer WM. Variation in number of hits for complex searches in Google Scholar. Journal of the Medical Library Association 2016; 104: 143-145.

Bramer WM, Giustini D, Kramer BM. Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study. Systematic Reviews 2016; 5: 39.

Bramer WM, Milic J, Mast F. Reviewing retrieved references for inclusion in systematic reviews using EndNote. Journal of the Medical Library Association 2017; 105: 84-87.

Bravata DM, McDonald KM, Shojania KG, Sundaram V, Owens DK. Challenges in systematic reviews: synthesis of topics related to the delivery, organization, and financing of health care. Ann Intern Med 2005; 142: 1056-1065.

Brietzke E, Gomes FA, Gerchman F, Freire RCR. Should systematic reviews and meta-analysis include data from preprints? Trends in Psychiatry and Psychotherapy 2023; 45: 1-2.

Briscoe S. A review of the reporting of web searching to identify studies for Cochrane systematic reviews. Research Synthesis Methods 2018; 9: 89-99.

Brody S, Brill R, Butcher R. Methods for Evaluating Database Coverage. Hypothesis: Research Journal for Health Information Professionals 2023; 35.

Brueton V, Tierney JF, Stenning S, Rait G. Identifying additional studies for a systematic review of retention strategies in randomised controlled trials: making contact with trials units and trial methodologists. Systematic reviews 2017; 6: 167.

Brunskill A, Hanneke R. The case of the disappearing librarians: analyzing documentation of librarians' contributions to systematic reviews. Journal of the Medical Library Association 2022; 110: 409-418.

Callaghan MW, Muller-Hansen F. Statistical stopping criteria for automated screening in systematic reviews. Systematic reviews 2020; 9: 273.

Callaway J. Librarian Reserve Corps: Literature indexing and metadata enhancement (LIME) observations from a year in the field. Journal of Health Information and Libraries Australasia 2021; 2: 35-45.

Campbell D, McDonald C, Cro S, Jairath V, Kahan BC. Access to unpublished protocols and statistical analysis plans of randomised trials. Trials 2022; 23: 674.

Cawley MA. Using Machine Learning to Locate Evidence More Efficiently: New Roles for Academic Research Librarians. 2022. In: Handbook of Research on Academic Libraries as Partners in Data Science Ecosystems [Internet]. Hershey, PA: IGI Global; [144-168]. Available from: https://www.igi-global.com/chapter/using-machine-learning-to-locate-evidence-more-efficiently/302752.

Center for Drug Evaluation and Research. Approval package for EpiPen Auto Injector and EpiPen Jr. Auto Injector. 1989. Available from: https://www.accessdata.fda.gov/drugsatfda_docs/nda/pre96/019430Orig1s001.pdf.

Centre for Reviews and Dissemination. Systematic Reviews: CRD's guidance for undertaking reviews in health care. York: University of York; 2009.

Chan AW. Out of sight but not out of mind: how to search for unpublished clinical trial evidence. BMJ 2012; 344: d8013.

Chapman SJ, Shelton B, Mahmood H, Fitzgerald JE, Harrison EM, Bhangu A. Discontinuation and non-publication of surgical randomised controlled trials: observational study. BMJ 2014; 349: g6870.

Chen KY, Borglund EM, Postema EC, Dunn AG, Bourgeois FT. Reporting of clinical trial safety results in ClinicalTrials.gov for FDA-approved drugs: A cross-sectional analysis. Clinical Trials 2022; 19: 442-451.

Chilcott J, Brennan A, Booth A, Karnon J, Tappenden P. The role of modelling in prioritising and planning clinical trials. HEALTH TECHNOLOGY ASSESSMENT 2003; 7: iii, 1-125.

Chow TK. Electronic search strategies should be repeatable. European Journal of Pain 2015; 19: 1562-1563.

Clyne B, Walsh KA, O'Murchu E, Sharp MK, Comber L, KK OB, Smith SM, Harrington P, O'Neill M, Teljeur C, Ryan M. Using preprints in evidence synthesis: Commentary on experience during the COVID-19 pandemic. J Clin Epidemiol 2021; 138: 203-210.

Cochrane Information Specialist Support Team. Section 3: The Cochrane Register of Studies (CRS). 2021a. In: Cochrane Information Specialists' Handbook [Internet]. Available from: https://training.cochrane.org/resource/cochrane-information-specialists-handbook.

Cochrane Information Specialist Support Team. Section 1: Role of a Cochrane Information Specialist. 2021b. In: Cochrane Information Specialists' Handbook [Internet]. Available from: https://training.cochrane.org/resource/cochrane-information-specialists-handbook.

Cochrane Information Specialist Support Team. Appendices. 2021c. In: Cochrane Information Specialists' Handbook [Internet]. Available from: https://training.cochrane.org/resource/cochrane-information-specialists-handbook.

Cochrane Style Manual Working Group. Cochrane Style Manual (accessed 28 April 2024). https://community.cochrane.org/style-manual.

Committee on Publication Ethics (COPE) Council. COPE Guidelines: Retraction guidelines. Version 2: November. 2019.  https://publicationethics.org/files/retraction-guidelines.pdf.

Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A, McIlroy W, Oxman AD. Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 1993; 269: 2749-2753.

Cooper C, Dawson S, Peters J, Varley-Campbell J, Cockcroft E, Hendon J, Churchill R. Revisiting the need for a literature search narrative: A brief methodological note. Research Synthesis Methods 2018a; 9: 361-365.

Cooper C, Varley-Campbell J, Booth A, Britten N, Garside R. Systematic review identifies six metrics and one method for assessing literature search effectiveness but no consensus on appropriate use. J Clin Epidemiol 2018b; 99: 53-63.

Cooper C, Booth A, Varley-Campbell J, Britten N, Garside R. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Medical Research Methodology 2018c; 18: 85.

Cooper C, Bou JT, Varley-Campbell J. Evaluating the effectiveness, efficiency, cost and value of contacting study authors in a systematic review: a case study and worked example. BMC Medical Research Methodology 2019; 19: 45.

Cooper C, Lorenc T, Schauberger U. What you see depends on where you sit: The effect of geographical location on web-searching for systematic reviews: A case study. Research Synthesis Methods 2021a; 12: 557-570.

Cooper C, Court R, Kotas E, Schauberger U. A technical review of three clinical trials register resources indicates where improvements to the search interfaces are needed. Research Synthesis Methods 2021b; 12: 384-393.

Cooper C, Dawson S, Lefebvre C. Searching for medical devices - Practical guidance. Research Synthesis Methods 2022a; 13: 144-154.

Cooper C, Booth A, Husk K, Lovell R, Frost J, Schauberger U, Britten N, Garside R. A tailored approach: A model for literature searching in complex systematic reviews. Journal of information science 2022b; 0.

Cooper H, Ribble RG. Influences on the outcome of literature searches for integrative research reviews. Science Communication 1989; 10: 179-201.

Coyne DW. The health-related quality of life was not improved by targeting higher hemoglobin in the Normal Hematocrit Trial. Kidney Int 2012; 82: 235-241.

Craven J, Levay P. Recording database searches for systematic reviews - What is the value of adding a narrative to peer-review checklists? A case study of NICE interventional procedures guidance. Evidence Based Library and Information Practice 2011; 6: 72-87.

Cullen L, Hanrahan K, Tucker S, Edmonds SW, Laures E. The Problem With the PICO Question: Shiny Object Syndrome and the PURPOSE Statement Solution. J Perianesth Nurs 2023; 38: 516-518.

Davidson M, Evrenoglou T, Graña C, Chaimani A, Boutron I. No evidence of important difference in summary treatment effects between COVID-19 preprints and peer-reviewed publications: a meta-epidemiological study. J Clin Epidemiol 2023; 162: 90-97.

de Kock S, Stirk L, Ross J, Duffy S, Noake C, Misso K. Systematic review search methods evaluated using the Preferred Reporting of Items for Systematic Reviews and Meta-Analyses and the Risk Of Bias In Systematic reviews tool. Int J Technol Assess Health Care 2020; 37: e18.

Decullier E, Huot L, Maisonneuve H. What time-lag for a retraction search on PubMed? BMC Research Notes 2014; 7: 395.

Dellavalle RP, Hester EJ, Heilig LF, Drake AL, Kuntzman JW, Graber M, Schilling LM. Information science. Going, going, gone: lost Internet references. Science 2003; 302: 787-788.

Dobrescu AI, Nussbaumer-Streit B, Klerings I, Wagner G, Persad E, Sommer I, Herkner H, Gartlehner G. Restricting evidence syntheses of interventions to English-language publications is a viable methodological shortcut for most medical topics: a systematic review. J Clin Epidemiol 2021; 137: 209-217.

Doshi P, Dickersin K, Healy D, Vedula SS, Jefferson T. Restoring invisible and abandoned trials: a call for people to publish the findings. BMJ 2013; 346: f2865.

Doshi P, Jefferson T. Open data 5 years on: a case series of 12 freedom of information requests for regulatory data to the European Medicines Agency. Trials 2016; 17: 78.

Doust JA, Pietrzak E, Sanders S, Glasziou PP. Identifying studies for systematic reviews of diagnostic tests was difficult due to the poor sensitivity and precision of methodologic filters and the lack of information in the abstract. J Clin Epidemiol 2005; 58: 444-449.

Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991; 337: 867-872.

Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Statistics in Medicine 2002; 21: 1635-1640.

Egger M, Zellweger Z, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet 1997; 350: 326-329.

Egger M, Smith GD. Bias in location and selection of studies. BMJ 1998; 316: 61-66.

Egger M, Jüni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technology Assessment 2003; 7: 1-76.

Egilman AC, Wallach JD, Morten CJ, Lurie P, Ross JS. Systematic overview of Freedom of Information Act requests to the Department of Health and Human Services from 2008 to 2017. Research Integrity and Peer Review 2019; 4: 26.

Egilman AC, Kapczynski A, McCarthy ME, Luxkaranayagam AT, Morten CJ, Herder M, Wallach JD, Ross JS. Transparency of Regulatory Data across the European Medicines Agency, Health Canada, and US Food and Drug Administration. J Law Med Ethics 2021; 49: 456-485.

Elia N, von Elm E, Chatagner A, Popping DM, Tramèr MR. How do authors of systematic reviews deal with research malpractice and misconduct in original studies? A cross-sectional analysis of systematic reviews and survey of their authors. BMJ Open 2016; 6: e010442.

Elsevier. Embase content coverage; 2024a. https://www.elsevier.com/solutions/embase-biomedical-research/coverage-and-content.

Elsevier. Embase resources; 2024b. https://www.elsevier.com/solutions/embase-biomedical-research/learn-and-support.

Enomoto M, Tseng CH, Hsu YC, Thuy LTT, Nguyen MH. Collaborating with AI in literature search-An important frontier. Hepatol Commun 2023; 7.

EUnetHTA JA3WP6B2-2 Authoring Team. Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness. Diemen (The Netherlands): EUnetHTA; 2019.  https://www.eunethta.eu/.

European Food Safety Authority. Application of systematic review methodology to food and feed safety assessments to support decision making. EFSA Journal 2010; 8: 1637.

Ewald H, Klerings I, Wagner G, Heise TL, Stratil JM, Lhachimi SK, Hemkens LG, Gartlehner G, Armijo-Olivo S, Nussbaumer-Streit B. Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study. J Clin Epidemiol 2022; 149: 154-164.

Eysenbach G, Trudel M. Going, going, still there: using the WebCite service to permanently archive cited web pages. Journal of Medical Internet Research 2005; 7: e60.

Faggion CM, Jr., Wu YC, Tu YK, Wasiak J. Quality of search strategies reported in systematic reviews published in stereotactic radiosurgery. Br J Radiol 2016; 89: 20150878.

Fain KM, Rajakannan T, Tse T, Williams RJ, Zarin DA. Results reporting for trials with the same sponsor, drug, and condition in ClinicalTrials.gov and peer-reviewed publications. JAMA Internal Medicine 2018; 178: 990-992.

Fanelli D, Wong J, Moher D. What difference might retractions make? An estimate of the potential epistemic cost of retractions on meta-analyses. Accountability in Research 2022; 29: 442-459.

Ferrante di Ruffano L, Davenport C, Eisinga A, Hyde C, Deeks JJ. A capture-recapture analysis demonstrated that randomized controlled trials evaluating the impact of diagnostic tests on patient outcomes are rare. J Clin Epidemiol 2012; 65: 282-287.

Fleminger J, Goldacre B. Prevalence of clinical trial status discrepancies: A cross-sectional study of 10,492 trials registered on both ClinicalTrials.gov and the European Union Clinical Trials Register. PLoS One 2018; 13: e0193088.

Folb BL, Klem ML, Youk AO, Dahm JJ, He M, Ketchum AM, Wessel CB, Hartman LM. Continuing education for systematic reviews: a prospective longitudinal assessment of a workshop for librarians. Journal of the Medical Library Association 2020; 108: 36-46.

Franco JVA, Garrote VL, Escobar Liquitay CM, Vietto V. Identification of problems in search strategies in Cochrane Reviews. Research Synthesis Methods 2018; 9: 408-416.

Frandsen TF, Gildberg FA, Tingleff EB. Searching for qualitative health research required several databases and alternative search strategies: a study of coverage in bibliographic databases. J Clin Epidemiol 2019a; 114: 118-124.

Frandsen TF, Eriksen MB, Hammer DMG, Christensen JB. PubMed coverage varied across specialties and over time: a large-scale study of included studies in Cochrane reviews. J Clin Epidemiol 2019b; 112: 59-66.

Frandsen TF, Bruun Nielsen MF, Lindhardt CL, Eriksen MB. Using the full PICO model as a search tool for systematic reviews resulted in lower recall for some PICO elements. J Clin Epidemiol 2020; 127: 69-75.

Frandsen TF, Nielsen MFB, Eriksen MB. Avoiding searching for outcomes called for additional search strategies: a study of Cochrane review searches. J Clin Epidemiol 2022; 149: 83-88.

Fraser N, Brierley L, Dey G, Polka JK, Palfy M, Nanni F, Coates JA. The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape. PLoS Biology 2021; 19: e3000959.

Furuya-Kanamori L, Lin L, Kostoulas P, Clark J, Xu C. Limits in the search date for rapid reviews of diagnostic test accuracy studies. Research Synthesis Methods 2023; 14: 173-179.

Garritty C, Gartlehner G, Nussbaumer-Streit B, King VJ, Hamel C, Kamel C, Affengruber L, Stevens A. Cochrane Rapid Reviews Methods Group offers evidence-informed guidance to conduct rapid reviews. J Clin Epidemiol 2021; 130: 13-22.

Gartlehner G, Affengruber L, Titscher V, Noel-Storr A, Dooley G, Ballarini N, Konig F. Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial. J Clin Epidemiol 2020; 121: 20-28.

Gaudino M, Robinson NB, Audisio K, Rahouma M, Benedetto U, Kurlansky P, Fremes SE. Trends and characteristics of retracted articles in the biomedical literature, 1971 to 2020. JAMA Internal Medicine 2021; 181: 1118-1121.

Ghezzi-Kopel K, Ault J, Chimwaza G, Diekmann F, Eldermire E, Gathoni N, Kelly J, Kinengyere AA, Kocher M, Lwoga ET, Page J, Young S, Porciello J. Making the case for librarian expertise to support evidence synthesis for the sustainable development goals. Research Synthesis Methods 2022; 13: 77-87.

Gianola S, Jesus TS, Bargeri S, Castellini G. Characteristics of academic publications, preprints, and registered clinical trials on the COVID-19 pandemic. PLoS One 2020; 15: e0240123.

Giroudon C, Armoiry X, Fieux M. Quality in systematic reviews: contribution of librarians 2024 (Accessed 14 February 2024). Available from: https://hal.science/hal-04441343,.

Glanville J, Lefebvre C, Manson P, Robinson S, Brbre I, Woods L, (editors). ISSG Search Filter Resource [Internet]. York (UK): The InterTASC Information Specialists' Sub-Group 2006 updated [20 May 2024, Cited 1 July 2024]. https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/home.

Glanville J, Foxlee R, Wisniewski S, Noel-Storr A, Edwards M, Dooley G. Translating the Cochrane EMBASE RCT filter from the Ovid interface to Embase.com: a case study. Health Information and Libraries Journal 2019; 36: 264-277.

Glanville JM, Duffy S, McCool R, Varley D. Searching ClinicalTrials.gov and the International Clinical Trials Registry Platform to inform systematic reviews: what are the optimal search approaches? Journal of the Medical Library Association 2014; 102: 177-183.

Godard-Sebillotte C, Le Berre M, Karunananthan S, Hong QN, Vedel I. A digital media strategy to obtain unpublished data for a systematic review yields a very high author response rate. J Clin Epidemiol 2018; 104: 141-143.

Godolphin PJ, Bath PM, Montgomery AA. Short email with attachment versus long email without attachment when contacting authors to request unpublished data for a systematic review: a nested randomised trial. BMJ Open 2019; 9: e025273.

Goldacre B, Lane S, Mahtani KR, Heneghan C, Onakpoya I, Bushfield I, Smeeth L. Pharmaceutical companies' policies on access to trial data, results, and methods: audit study. BMJ 2017; 358: j3334.

Grabenstein D. The state of global clinical trial disclosure: what noncompliance penalties are in place, and how they are enforced [White paper]. 2023.  https://www.clinicalleader.com/doc/the-state-of-global-clinical-trial-disclosure-what-noncompliance-penalties-are-in-place-and-how-they-are-enforced-0001.

Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 2005; 331: 1064-1065.

Gresham G, Meinert JL, Gresham AG, Piantadosi S, Meinert CL. Update on the clinical trial landscape: analysis of ClinicalTrials.gov registration data, 2000-2020. Trials 2022; 23: 858.

Guimarães NS, Joviano-Santos JV, Reis MG, Chaves RRM. Development of search strategies for systematic reviews in health using ChatGPT: a critical analysis. J Transl Med 2024; 22: 1.

Haddaway NR, Collins AM, Coughlin D, Kirk S. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PLoS One 2015; 10: e0138237.

Haddaway NR, Rethlefsen ML, Davies M, Glanville J, McGowan B, Nyhan K, Young S. A suggested data structure for transparent and repeatable reporting of bibliographic searching. Campbell Syst Rev 2022; 18: e1288.

Halfpenny NJ, Quigley JM, Thompson JC, Scott DA. Value and usability of unpublished data sources for systematic reviews and network meta-analyses. Evidence-based Medicine 2016; 21: 208-213.

Halladay CW, Trikalinos TA, Schmid IT, Schmid CH, Dahabreh IJ. Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions. J Clin Epidemiol 2015; 68: 1076-1084.

Hameed I, Demetres M, Tam DY, Rahouma M, Khan FM, Wright DN, Mages K, DeRosa AP, Baltich Nelson B, Pain K, Delgado D, Girardi LN, Fremes SE, Gaudino M. An assessment of the quality of current clinical meta-analyses. BMC Medical Research Methodology 2020; 20: 105.

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Medical Research Methodology 2016; 16: 127.

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Medical Research Methodology 2017; 17: 64.

Hateley E. “We need to talk about PICO”: Approaching the move from question formulation to search strategy. Journal of Health Information and Libraries Australasia 2023; 4: 31-38.

Hausner E. Problems encountered with ICTRP Search Portal. Comment on: van Enst WA, Scholten RJPM, Hooft L. Identification of additional trials in prospective trial registers for Cochrane Systematic Reviews. PLoS One 2014; 7: e42812.

Heron L, Buitrago-Garcia D, Ipekci AM, Baumann R, Imeri H, Salanti G, Counotte MJ, Low N. How to update a living systematic review and keep it alive during a pandemic: a practical guide. Systematic reviews 2023; 12: 156.

Hetherington J, Dickersin K, Chalmers I, Meinert CL. Retrospective and prospective identification of unpublished controlled trials: lessons from a survey of obstetricians and pediatricians. Pediatrics 1989; 84: 374-380.

Hinde S, Spackman E. Bidirectional citation searching to completion: an exploration of literature searching methods. Pharmacoeconomics 2015; 33: 5-11.

Hirt J, Nordhausen T, Fuerst T, Ewald H, Appenzeller-Herzog C. Guidance on terminology, application, and reporting of citation searching: the TARCiS statement. BMJ 2024a; 385: e078384.

Hirt J, Ewald H, Briel M, Schandelmaier S. Searching a methods topic: practical challenges and implications for search design. J Clin Epidemiol 2024b; 166: 111201.

Hodkinson A, Dietz KC, Lefebvre C, Golder S, Jones M, Doshi P, Heneghan C, Jefferson T, Boutron I, Stewart L. The use of clinical study reports to enhance the quality of systematic reviews: a survey of systematic review authors. Syst Rev 2018; 7: 117.

Horton R. Medical editors trial amnesty. Lancet 1997; 350: 756.

Hoy MB. Rise of the Rxivs: How preprint servers are changing the publishing process. Medical Reference Services Quarterly 2020; 39: 84-89.

Hunter KE, Webster AC, Page MJ, Willson M, McDonald S, Berber S, Skeers P, Tan-Koay AG, Parkhill A, Seidler AL. Searching clinical trials registers: guide for systematic reviewers. BMJ 2022; 377: e068791.

Hwang TJ, Carpenter D, Lauffenburger JC, Wang B, Franklin JM, Kesselheim AS. Failure of investigational drugs in late-stage clinical development and publication of trial results. JAMA Internal Medicine 2016; 176: 1826-1833.

Iacobucci G. Nearly half of all trials run by major sponsors in past decade are unpublished. BMJ 2016; 355: i5955.

Iansavichene AE, Sampson M, McGowan J, Ajiferuke IS. Should systematic reviewers search for randomized, controlled trials published as letters? Ann Intern Med 2008; 148: 714-715.

Institute for Quality and Efficiency in Health Care. IQWiG general methods: Version 7.0. Cologne; 2023.  https://www.iqwig.de/methoden/general-methods_version-7-0.pdf.

Institute of Medicine. Finding what works in health care: Standards for systematic reviews. Washington, DC: The National Academies Press; 2011.  http://books.nap.edu/openbook.php?record_id=13059.

Irvin E, Hayden J. Developing and testing an optimal search strategy for identifying studies of prognosis [Poster] 14th Cochrane Colloquium, Dublin, Ireland; October 23-26 2006. https://abstracts.cochrane.org/2006-dublin/developing-and-testing-optimal-search-strategy-identifying-studies-prognosis.

Isojarvi J, Wood H, Lefebvre C, Glanville J. Challenges of identifying unpublished data from clinical trials: getting the best out of clinical trials registers and other novel sources. Research Synthesis Methods 2018: 561-578.

Isojarvi J, Glanville J. Evidence-based searching for health technology assessment: keeping up to date with SuRe Info. International Journal of Technology Assessment in Health Care 2021; 37: e51.

Itani D, Lababidi G, Itani R, El Ghoul T, Hamade L, Hijazi ARA, Khabsa J, Akl EA. Reporting of funding and conflicts of interest improved from preprints to peer-reviewed publications of biomedical research. J Clin Epidemiol 2022; 149: 146-153.

Jefferson T, Doshi P, Boutron I, Golder S, Heneghan C, Hodkinson A, Jones M, Lefebvre C, Stewart LA. When to include clinical study reports and regulatory documents in systematic reviews. BMJ Evidence-Based Medicine 2018; 23: 210-217.

Jia Y, Huang D, Wen J, Wang Y, Rosman L, Chen Q, Robinson KA, Gagnier JJ, Ehrhardt S, Celentano DD. Assessment of language and indexing biases among Chinese-sponsored randomized clinical trials. JAMA Network Open 2020; 3: e205894.

Jiao S, Tsutani K, Haga N. Review of Cochrane reviews on acupuncture: how Chinese resources contribute to Cochrane reviews. Journal of Alternative and Complementary Medicine 2013; 19: 613-621.

Jørgensen L, Gøtzsche PC, Jefferson T. Index of the human papillomavirus (HPV) vaccine industry clinical study programmes and non-industry funded studies: a necessary basis to address reporting bias in a systematic review. Syst Rev 2018; 7: 8.

Jüni P, Holenstein F, Sterne J, Bartlett C, Egger M. Direction and impact of language bias in meta-analyses of controlled trials: empirical study. International journal of epidemiology 2002; 31: 115-123.

Kastner M, Straus SE, McKibbon KA, Goldsmith CH. The capture-mark-recapture technique can be used as a stopping rule when searching in systematic reviews. J Clin Epidemiol 2009; 62: 149-157.

Kataoka Y, Banno M, Tsujimoto Y, Ariie T, Taito S, Suzuki T, Oide S, Furukawa TA. Retracted randomized controlled trials were cited and not corrected in systematic reviews and clinical practice guidelines. J Clin Epidemiol 2022; 150: 90-97.

Kaul T, Colombijn JMT, Vernooij RWM, Spijker R, Idema DL, Huis In 't Veld LF, Damen JAA, Hooft L. Both clinical trial register and electronic bibliographic database searches were needed to identify randomized clinical trials for systematic reviews: an evaluation study. J Clin Epidemiol 2024; 169: 111300.

Khademizadeh S, Danesh F, Esmaeili S, Lund B, Santos-d'Amorim K. Evolution of retracted publications in the medical sciences: Citations analysis, bibliometrics, and altmetrics trends. Account Res 2023: 1-16.

Kirkham JJ, Penfold NC, Murphy F, Boutron I, Ioannidis JP, Polka J, Moher D. Systematic examination of preprint platforms for use in the medical and biomedical sciences setting. BMJ Open 2020; 10: e041849.

Knelangen M, Hausner E, Metzendorf M-I, Sturtz S, Waffenschmidt S. Trial registry searches for randomized controlled trials of new drugs required registry-specific adaptation to achieve adequate sensitivity. J Clin Epidemiol 2018; 94: 69-75.

Koffel JB. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors. PLoS One 2015; 10: e0125931.

Kohl CBS, Faggion CM, Jr. A comprehensive overview of studies that assessed article retractions within the biomedical sciences. Account Res 2024; 31: 557-575.

Köhler M, Haag S, Biester K, Brockhaus AC, McGauran N, Grouven U, Kölsch H, Seay U, Hörn H, Moritz G, Staeck K, Wieseler B. Information on new drugs at market entry: retrospective analysis of health technology assessment reports versus regulatory reports, journal publications, and registry reports. BMJ 2015; 350: h796.

Kreis J, Panteli D, Busse R. How health technology assessment agencies address the issue of unpublished data. INTERNATIONAL JOURNAL OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 2014; 30: 34-43.

Lampert A, Hoffmann GF, Ries M. Ten years after the International Committee of Medical Journal Editors' clinical trial registration initiative, one quarter of phase 3 pediatric epilepsy clinical trials still remain unpublished: a cross sectional analysis. PLoS One 2016; 11: e0144973.

Lane D, Dykeman J, Ferri M, Goldsmith CH, Stelfox HT. Capture-mark-recapture as a tool for estimating the number of articles available for systematic reviews in critical care medicine. Journal of Critical Care 2013; 28: 469-475.

Layton D. A critical review of search strategies used in recent systematic reviews published in selected prosthodontic and implant-related journals: Are systematic reviews actually systematic? International Journal of Prosthodontics 2017; 30: 13-21.

Lê M-L, Neilson CJ, Winkler J. [Preprint]. Benchmarking Librarian Support of Systematic Reviews in the Sciences, Humanities, and Social Sciences 2023 [11 July 2023]. Available from: https://osf.io/v7m9y/.

Lee K, Bacchetti P, Sim I. Publication of clinical trials supporting successful new drug applications: a literature analysis. PLoS Medicine 2008; 5: e191.

Lefebvre C, Eisinga A, McDonald S, Paul N. Enhancing access to reports of randomized trials published world-wide--the contribution of EMBASE records to the Cochrane Central Register of Controlled Trials (CENTRAL) in The Cochrane Library. Emerging Themes in Epidemiology 2008; 5: 13.

Lefebvre C, Glanville J, Wieland LS, Coles B, Weightman AL. Methodological developments in searching for studies for systematic reviews: past, present and future? Systematic reviews 2013; 2: 78.

Lefebvre C, Glanville J, Beale S, Boachie C, Duffy S, Fraser C, Harbour J, McCool R, Smith L. Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study. HEALTH TECHNOLOGY ASSESSMENT 2017; 21: 1-148.

Lefebvre C, Duffy S. Peer review of searches for studies for health technology assessments, systematic reviews, and other evidence syntheses. INTERNATIONAL JOURNAL OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 2021; 37: e64.

Levay P, Heath A, Tuvey D. Efficient searching for NICE public health guidelines: Would using fewer sources still find the evidence? Research Synthesis Methods 2022; 13: 760-789.

Li D, Kanoulas E. When to Stop Reviewing in Technology-Assisted Reviews: Sampling from an Adaptive Distribution to Estimate Residual Relevant Documents. ACM Transactions on Information Systems 2020; 38: Article 41.

Liu S, Bourgeois FT, Dunn AG. Identifying unreported links between ClinicalTrials.gov trial registrations and their published results. Research Synthesis Methods 2022; 13: 342-352.

Lorenzetti DL, Topfer LA, Dennett L, Clement F. Value of databases other than MEDLINE for rapid health technology assessments. INTERNATIONAL JOURNAL OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 2014; 30: 173-178.

MacDonald H, Comer C, Foster M, Labelle PR, Marsalis S, Nyhan K, Premji Z, Rogers M, Splenda R, Stansfield C, Young S. Searching for studies: A guide to information retrieval for Campbell systematic reviews. Campbell Syst Rev 2024; 20: e1433.

MacLean CH, Morton SC, Ofman JJ, Roth EA, Shekelle PG, Southern California Evidence-Based Practice Center. How useful are unpublished data from the Food and Drug Administration in meta-analysis? J Clin Epidemiol 2003; 56: 44-51.

Manheimer E, Anderson D. Survey of public information about ongoing clinical trials funded by industry: evaluation of completeness and accessibility. BMJ 2002; 325: 528-531.

Marshall C, Brereton P. Systematic review toolbox: a catalogue of tools to support systematic reviews. Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering (EASE) 2015: Article no. 23.

Marshall I, Noel-Storr A, Kuiper J, Thomas J, Wallace BC. Machine learning for identifying randomized controlled trials: an evaluation and practitioner's guide. Research Synthesis Methods 2018a; 9: 602-614.

Marshall IJ, Noel-Storr A, Kuiper J, Thomas J, Wallace BC. Machine learning for identifying Randomized Controlled Trials: an evaluation and practitioner's guide. Research Synthesis Methods 2018b; 9: 602-614.

Marshall IJ, Trikalinos TA, Soboczenski F, Yun HS, Kell G, Marshall R, Wallace BC. In a pilot study, automated real-time systematic review updates were feasible, accurate, and work-saving. J Clin Epidemiol 2023; 153: 26-33.

Masterson D, Martinez-Silveira MS. Aplicação do Peer Review of Electronic Search Strategies para avaliação da qualidade das estratégias de busca das revisões sistemáticas [Application of Peer Review of Electronic Search Strategies (PRESS) to assess the quality of systematic reviews search strategies]. Em Questão 2022; 28: 117865.

Matthews EJ, Edwards AG, Barker J, Bloor M, Covey J, Hood K, Pill R, Russell I, Stott N, Wilkinson C. Efficient literature searching in diffuse topics: lessons from a systematic review of research on communicating risk to patients in primary care. Health Libraries Review 1999; 16: 112-120.

McDonagh MS, Thakurta S, Peterson K. A study of the value of requesting information from drug manufacturers for systematic reviews; 9 years of experience from the drug effectiveness review project. Systematic reviews 2018; 7: 172.

McGowan J, Sampson M, Salzwedel D, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Explanation and Elaboration (PRESS E&E) Ottawa: CADTH; 2016a.  www.cda-amc.ca/sites/default/files/pdf/CP0015_PRESS_Update_Report_2016.pdf.

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. Journal of Clinical Epidemiology 2016b; 75: 40-46.

Meert D, Torabi N, Costella J. Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. Journal of the Medical Library Association 2016; 104: 267-277.

Metzendorf M-I. Why medical information specialists should routinely form part of teams producing high quality systematic reviews – a Cochrane perspective. Journal of the European Association for Health Information and Libraries 2016; 12: 6-9.

Metzendorf MI, Featherstone RM. Frandsen et al. provide insights to PubMed coverage across all Cochrane Review Groups, but more in-depth analyses would further help inform database choices. J Clin Epidemiol 2019; 116: 135-136.

Meursinge Reynders R, Ladu L, Di Girolamo N. Contacting of authors modified crucial outcomes of systematic reviews but was poorly reported, not systematic, and produced conflicting results. J Clin Epidemiol 2019; 115: 64-76.

Mirjam Keestra S, Rodgers F, Gepp S, Grabitz P, Bruckner T. Improving clinical trial transparency at UK universities: Evaluating 3 years of policies and reporting performance on the European Clinical Trial Register. Clin Trials 2022; 19: 217-223.

Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. HEALTH TECHNOLOGY ASSESSMENT 2003; 7: 1-90.

Morrison A, Polisena J, Husereau D, Moulton K, Clark M, Fiander M, Mierzwinski-Urban M, Clifford T, Hutton B, Rabb D. The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. INTERNATIONAL JOURNAL OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 2012; 28: 138-144.

Mott A, Fairhurst C, Torgerson D. Assessing the impact of retraction on the citation of randomized controlled trial reports: an interrupted time-series analysis. J Health Serv Res Policy 2019; 24: 44-51.

Muller AE, Ames H, Himmels J, Jardim PJ, Nguyen L, Rose C, Van de Velde S. Implementation of machine learning in evidence syntheses in the Cluster for Reviews and Health Technology Assessments: Final report 2020-2021. Oslo; 2021.

Mullins MM, DeLuca JB, Crepaz N, Lyles CM. Reporting quality of search methods in systematic reviews of HIV behavioral interventions (2000–2010): are the searches clearly explained, systematic and reproducible? Research Synthesis Methods 2014; 5: 116-130.

National Institute for Health and Care Excellence (NICE). Identifying the evidence: literature searching and evidence submission. 2014. In: Developing NICE guidelines: the manual (Last updated: 17 January 2024) Process and Methods Guides 20 [Internet]. Available from: https://www.nice.org.uk/process/pmg20/chapter/identifying-the-evidence-literature-searching-and-evidence-submission.

Neilson CJ. Adoption of peer review of literature search strategies in knowledge synthesis from 2009 to 2018: An overview. Health Information & Libraries Journal 2021; 38: 160-171.

Nelson JT, Tse T, Puplampu-Dove Y, Golfinopoulos E, Zarin DA. Comparison of availability of trial results in ClinicalTrials.gov and PubMed by data source and funder type. JAMA 2023; 329: 1404-1406.

Nguyen PY, Kanukula R, McKenzie JE, Alqaidoom Z, Brennan SE, Haddaway NR, Hamilton DG, Karunananthan S, McDonald S, Moher D, Nakagawa S, Nunan D, Tugwell P, Welch VA, Page MJ. Changing patterns in reporting and sharing of review data in systematic reviews with meta-analysis of the effects of interventions: cross sectional meta-research study. BMJ 2022; 379: e072428.

Niederstadt C, Droste S. Reporting and presenting information retrieval processes: the need for optimizing common practice in health technology assessment. INTERNATIONAL JOURNAL OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 2010; 26: 450-457.

Noel-Storr A, Dooley G, Elliott J, Steele E, Shemilt I, Mavergames C, Wisniewski S, McDonald S, Murano M, Glanville J, Foxlee R, Beecher D, Ware J, Thomas J. An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials. J Clin Epidemiol 2021a; 133: 130-139.

Noel-Storr A, Dooley G, Affengruber L, Gartlehner G. Citation screening using crowdsourcing and machine learning produced accurate results: Evaluation of Cochrane's modified Screen4Me service. J Clin Epidemiol 2021b; 130: 23-31.

Noel-Storr A, Wisniewski S. Evaluation of Cochrane’s Evidence Pipeline for populating CENTRAL. Global Evidence Summit; 2024 September 10-13, 2024; Czech Republic, Prague. https://programme.ges24.myeventapp.com/.

Noel-Storr AH, Dooley G, Wisniewski S, Glanville J, Thomas J, Cox S, Featherstone R, Foxlee R. Cochrane Centralised Search Service showed high sensitivity identifying randomized controlled trials: A retrospective analysis. J Clin Epidemiol 2020; 127: 142-150.

Nussbaumer-Streit B, Klerings I, Wagner G, Heise TL, Dobrescu AI, Armijo-Olivo S, Stratil JM, Persad E, Lhachimi SK, Van Noord MG, Mittermayr T, Zeeb H, Hemkens L, Gartlehner G. Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study. J Clin Epidemiol 2018; 102: 1-11.

Nussbaumer-Streit B, Klerings I, Dobrescu AI, Persad E, Stevens A, Garritty C, Kamel C, Affengruber L, King VJ, Gartlehner G. Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study. J Clin Epidemiol 2020; 118: 42-54.

O'Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic Reviews 2015; 4: 5. (Erratum in: Systematic Reviews 2015; 2014: 2059).

Oikonomidi T, Boutron I, Pierre O, Cabanac G, Ravaud P, Covid-NMA Consortium. Changes in evidence for studies assessing interventions for COVID-19 reported in preprints: meta-research study. BMC Medicine 2020; 18: 402.

Oxman AD, Guyatt GH. The science of reviewing research. Annals of the New York Academy of Sciences 1993; 703: 125-133; discussion 133-134.

Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hrobjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, McKenzie JE. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ 2021a; 372: n160.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hrobjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021b; 372: n71.

Paladin I, Pranić SM. Reporting of the safety from allergic rhinitis trials registered on ClinicalTrials.gov and in publications: An observational study. BMC Medical Research Methodology 2022; 22: 262.

Pansieri C, Pandolfini C, Bonati M. Clinical trial registries: more international, converging efforts are needed. Trials 2017; 18: 86.

Pauwels NS, Koobasi M, Fry A, Vandendriessche T, Wittevrongel A, Ødegaard M. Guidance for systematic reviews in journal author instructions: Findings and recommendations for editorial teams. Cochrane Evidence Synthesis and Methods 2024; 2: e12050.

Pawliuk C, Cheng S, Zheng A, Siden HH. Librarian involvement in systematic reviews was associated with higher quality of reported search methods: a cross-sectional survey. J Clin Epidemiol 2024; 166: 111237.

Pham B, Klassen TP, Lawson ML, Moher D. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol 2005; 58: 769-776.

Pieper D, Puljak L. Language restrictions in systematic reviews should not be imposed in the search strategy but in the eligibility criteria if necessary. J Clin Epidemiol 2021; 132: 146-147.

Pilkington K, Boshnakova A, Clarke M, Richardson J. "No language restrictions" in database searches: what does this really mean? Journal of Alternative and Complementary Medicine 2005; 11: 205-207.

Price C, Saragossi J, Tran C, Lyon J. [Preprint]. Evaluating the consistency and quality of search strategies and methodology in Cochrane Urology Group Systematic Reviews2022 15 August 2023. Available from: https://osf.io/28u6z.

Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Research Synthesis Methods 2014; 5: 98-115.

Ramirez D, Foster MJ, Kogut A, Xiao D. Adherence to systematic review standards: Impact of librarian involvement in Campbell Collaboration's education reviews. The Journal of Academic Librarianship 2022; 48: 102567.

Relevo R, Paynter R. Peer Review of Search Strategies. Rockville (MD): Agency for Healthcare Research and Quality (US); 2012.  https://www.ncbi.nlm.nih.gov/books/NBK98353/.

Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol 2015; 68: 617-626.

Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB, PRISMA-S. Group. PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. Systematic reviews 2021; 10: 39.

Rethlefsen ML, Page MJ. PRISMA 2020 and PRISMA-S: common questions on tracking records and the flow diagram. Journal of the Medical Library Association 2022; 110: 253-257.

Rethlefsen ML, Brigham TJ, Price C, Moher D, Bouter LM, Kirkham JJ, Schroter S, Zeegers MP. Systematic review search strategies are poorly reported and not reproducible: a cross-sectional metaresearch study. J Clin Epidemiol 2024; 166: 111229.

Reveiz L, Cardona AF, Ospina EG, de Agular S. An e-mail survey identified unpublished studies for systematic reviews. J Clin Epidemiol 2006; 59: 755-758.

Rice DB, Kloda LA, Levis B, Qi B, Kingsland E, Thombs BD. Are MEDLINE searches sufficient for systematic reviews and meta-analyses of the diagnostic accuracy of depression screening tools? A review of meta-analyses. Journal of Psychosomatic Research 2016; 87: 7-13.

Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions. ACP Journal Club 1995; 123: A12-13.

Ross-White A. An environmental scan of librarian involvement in systematic reviews at Queen's University: 2020 update. Journal of the Canadian Health Libraries Association / Journal de l'Association des bibliothèques de la santé du Canada 2021; 42: 110-117.

Roundtree AK, Kallen MA, Lopez-Olivo MA, Kimmel B, Skidmore B, Ortiz Z, Cox V, Suarez-Almazor ME. Poor reporting of search strategy and conflict of interest in over 250 narrative and systematic reviews of two biologic agents in arthritis: a systematic review. J Clin Epidemiol 2009; 62: 128-137.

Royle P, Milne R. Literature searching for randomized controlled trials used in Cochrane reviews: rapid versus exhaustive searches. INTERNATIONAL JOURNAL OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 2003; 19: 591-603.

Royle P, Waugh N. Should systematic reviews include searches for published errata? Health Information & Libraries Journal 2004; 21: 14-20.

Salvador-Olivan JA, Marco-Cuenca G, Arquero-Aviles R. Errors in search strategies used in systematic reviews and their effects on information retrieval. Journal of the Medical Library Association 2019; 107: 210-221.

Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, St John PD, Viola R, Raina P. Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol 2003; 56: 943-955.

Sampson M, Zhang L, Morrison A, Barrowman NJ, Clifford TJ, Platt RW, Klassen TP, Moher D. An alternative to the hand searching gold standard: Validating methodological search filters using relative recall. BMC Medical Research Methodology 2006; 6: 33.

Sampson M, McGowan J. Errors in search strategies were identified by type and frequency. J Clin Epidemiol 2006; 59: 1057-1063.

Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting methods for systematic reviews. J Clin Epidemiol 2008; 61: 748-754.

Sampson M, McGowan J. Inquisitio validus Index Medicus: A simple method of validating MEDLINE systematic review searches. Research Synthesis Methods 2011; 2: 103-109.

Sampson M, de Bruijn B, Urquhart C, Shojania K. Complementary approaches to searching MEDLINE may be sufficient for updating systematic reviews. Journal of Clinical Epidemiology 2016; 78: 108-115.

Savoie I, Helmer D, Green CJ, Kazanjian A. Beyond Medline: reducing bias through extended systematic review search. Int J Technol Assess Health Care 2003; 19: 168-178.

Scells H, Zuccon G. searchrefiner: A Query Visualisation and Understanding Tool for Systematic Reviews.  Proceedings of the 27th ACM International Conference on Information and Knowledge Management; Torino, Italy: Association for Computing Machinery; 2018. p. 1939–1942. doi: 1910.1145/3269206.3269215.

Schellinger J, Sewell K, Bloss JE, Ebron T, Forbes C. The effect of librarian involvement on the quality of systematic reviews in dental medicine. PLoS One 2021; 16: e0256833.

Scherer RW, Ugarte-Gil C, Schmucker C, Meerpohl JJ. Authors report lack of time as main reason for unpublished research presented at biomedical conferences: a systematic review. J Clin Epidemiol 2015; 68: 803-810.

Schroll JB, Bero L, Gøtzsche PC. Searching for unpublished data for Cochrane reviews: cross sectional study. BMJ 2013; 346: f2231.

Schvaneveldt N, Stellrecht EM. Assessing the roles and challenges of librarians in dental systematic and scoping reviews. J Med Libr Assoc 2021; 109: 52-61.

Shamsi M, Mirzaei M, Vaziri S, Mozaffari HR. Common mistakes in reporting systematic reviews and meta-analyses. Health Promot Perspect 2020; 10: 97.

Shemilt I, Simon A, Hollands GJ, Marteau TM, Ogilvie D, O'Mara-Eves A, Kelly MP, Thomas J. Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Research Synthesis Methods 2014; 5: 31-49.

Shemilt I, Khan N, Park S, Thomas J. Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Systematic reviews 2016; 5: 140.

Shemilt I, Arno A, Thomas J, Lorenc T, Khouja C, Raine G, Sutcliffe K, Preethy DS, Kwan I, Wright K, Sowden A. Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research [version 1; peer review: 2 approved with reservations]2021; (6):[210 p.].

Shemilt I, Noel-Storr A, Thomas J, Featherstone R, Mavergames C. Machine learning reduced workload for the Cochrane COVID-19 Study Register: development and evaluation of the Cochrane COVID-19 Study Classifier. Syst Rev 2022; 11: 15.

Smalheiser NR, Holt AW. A web-based tool for automatically linking clinical trials to their publications. Journal of the American Medical Informatics Association 2022; 29: 822-830.

Sneyd A, Stevenson M. Modelling stopping criteria for search results using Poisson processes. 2019. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) [Internet]. [3484–3489]. Available from: https://aclanthology.org/D19-1351.

Speich B, Gloy VL, Klatte K, Gryaznov D, Taji Heravi A, Ghosh N, Marian IR, Lee H, Mansouri A, Lohner S, Saccilotto R, Nury E, Chan AW, Blümle A, Odutayo A, Hopewell S, Briel M. Reliability of Trial Information Across Registries for Trials With Multiple Registrations: A Systematic Review. JAMA Netw Open 2021; 4: e2128898.

Speich B, Gryaznov D, Busse JW, Gloy VL, Lohner S, Klatte K, Taji Heravi A, Ghosh N, Lee H, Mansouri A, Marian IR, Saccilotto R, Nury E, Kasenda B, Ojeda-Ruiz E, Schandelmaier S, Tomonaga Y, Amstutz A, Pauli-Magnus C, Bischoff K, Wollmann K, Rehner L, Meerpohl JJ, Nordmann A, Wong J, Chow N, Hong PJ, Mc Cord-De Iaco K, Sricharoenchai S, Agarwal A, Schwenkglenks M, Hemkens LG, von Elm E, Copsey B, Griessbach AN, Schönenberger C, Mertz D, Blümle A, von Niederhäusern B, Hopewell S, Odutayo A, Briel M. Nonregistration, discontinuation, and nonpublication of randomized trials: A repeated metaresearch analysis. PLoS Med 2022; 19: e1003980.

Spencer AJ, Eldredge JD. Roles for librarians in systematic reviews: a scoping review. Journal of the Medical Library Association 2018; 106: 46-56.

Spijker R, Dinnes J, Glanville J, Eisinga A. Chapter 6: Searching for and selecting studies. 2023. In: Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 20 (updated July 2023) [Internet]. Cochrane. Available from: https://training.cochrane.org/handbook-diagnostic-test-accuracy/current.

Spoor P, Airey M, Bennett C, Greensill J, Williams R. Use of the capture-recapture technique to evaluate the completeness of systematic literature searches. BMJ 1996; 313: 342-343.

Spry C, Mierzwinski-Urban M, Rabb D. Peer review of literature search strategies: does it make a difference? 21st Cochrane Colloquium; 2013; Quebec City, Canada. https://abstracts.cochrane.org/2013-quebec-city/peer-review-literature-search-strategies-does-it-make-difference.

Spry C, Mierzwinski-Urban M. The impact of the peer review of literature search strategies in support of rapid review reports. Research Synthesis Methods 2018; 9: 521-526.

Stansfield C, Stokes G, Thomas J. Applying machine classifiers to update searches: Analysis from two case studies. Research Synthesis Methods 2022; 13: 121-133.

Stansfield C, Clowes M, Booth A, Thomas J. Chapter 5. Searching and identifying studies. 2024. In: Cochrane-Campbell Handbook for Qualitative Evidence Synthesis [Internet]. London: Cochrane. Available from: https://training.cochrane.org/qeschapter5serv0240823.

Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complementary Therapies in Medicine 2004; 12: 228-232.

Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Controlled Clinical Trials 2000; 21: 476-487.

Suelzer EM, Deal J, Hanus K, Ruggeri BE, Witkowski E. Challenges in Identifying the Retracted Status of an Article. JAMA Netw Open 2021; 4: e2115648.

Svarre T, Russell-Rose T. Think outside the search box: A comparative study of visual and form-based query builders. Journal of information science 2022: DOI: 10.1177/01655515221138536.

The Methods Group of the Campbell Collaboration. Methodological expectations of Campbell Collaboration intervention reviews: Conduct standards: Campbell Collaboration; 2019. Available from: https://onlinelibrary.wiley.com/page/journal/18911803/homepage/author-guidelines.

Thomas J, Noel-Storr A, Marshall I, Wallace B, McDonald S, Mavergames C, Glasziou P, Shemilt I, Synnot A, Turner T, Elliott J, Living Systematic Review Network. Living systematic reviews: 2. Combining human and machine effort. J Clin Epidemiol 2017; 91: 31-37.

Thomas J, McDonald S, Noel-Storr A, Shemilt I, Elliott J, Mavergames C, Marshall IJ. Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews. J Clin Epidemiol 2021; 133: 140-151.

Thomas J. Evidence surveillance during the COVID-19 pandemic using automation and crowdsourcing. Journal of EAHIL 2021; 17: 5-6.

Tramèr MR, Reynolds DJ, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis: a case study. BMJ 1997; 315: 635-640.

Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Systematic reviews 2014; 3: 74.

Tsujimoto Y, Tsutsumi Y, Kataoka Y, Banno M, Furukawa TA. Around ten percent of most recent Cochrane reviews included outcomes in their literature search strategy and were associated with potentially exaggerated results: A research-on-research study. J Clin Epidemiol 2022; 141: 74-81.

US National Library of Medicine. MEDLINE, PubMed, and PMC (PubMed Central): How are they different? (Last Reviewed: December 28  2023). https://www.nlm.nih.gov/bsd/difference.html.

US National Library of Medicine. MEDLINE: Overview (Last reviewed: February 5 2024). https://www.nlm.nih.gov/medline/medline_overview.html.

US National Library of Medicine. PubMed Overview (Last update: August 15, 2023). 2023. https://pubmed.ncbi.nlm.nih.gov/about/.

Uttley L, Quintana DS, Montgomery P, Carroll C, Page MJ, Falzon L, Sutton A, Moher D. The problems with systematic reviews: a living systematic review. J Clin Epidemiol 2023; 156: 30-41.

Viergever RF, Li K. Trends in global clinical trial registration: an analysis of numbers of registered clinical trials in different parts of the world from 2004 to 2013. BMJ Open 2015; 5: e008932.

von Elm E, Poglia G, Walder B, Tramèr MR. Different patterns of duplicate publication: an analysis of articles used in systematic reviews. JAMA 2004; 291: 974-980.

Waffenschmidt S, Knelangen M, Sieben W, Buhn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Medical Research Methodology 2019; 19: 132.

Wallace S, Daly C, Campbell M, Cody J, Grant A, Vale L, Donaldson C, Khan I, Lawrence P, MacLeod A. After MEDLINE? Dividend from other potential sources of randomised controlled trials. Second International Conference Scientific Basis of Health Services & Fifth Annual Cochrane Colloquium; 1997; Amsterdam, The Netherlands. https://abstracts.cochrane.org/1997-amsterdam/after-medline-dividend-other-potential-sources-randomised-controlled-trials.

Wang S, Scells H, Koopman B, Zuccon G. Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search? arXiv 2023: 1426-1436.

Wasiak J, Shen AY, Ware R, O'Donohoe TJ, Faggion CM, Jr. Methodological quality and reporting of systematic reviews in hand and wrist pathology. J Hand Surg Eur Vol 2017; 42: 852-856.

Weber EJ, Callaham ML, Wears RL, Barton C, Young G. Unpublished research from a medical specialty meeting: why investigators fail to publish. JAMA 1998; 280: 257-259.

Wieseler B, Kerekes MF, Vervoelgyi V, McGauran N, Kaiser T. Impact of document type on reporting quality of clinical drug trials: a comparison of registry reports, clinical study reports, and journal publications. BMJ 2012; 344: d8141.

Witkowski MA, Aldhouse N. Transparency and reproducibility of supplementary search methods in NICE single technology appraisal manufacturer submissions. Value in Health 2015; 18: A721-722.

Wolfe N, Gotzsche PC, Bero L. Strategies for obtaining unpublished drug trial data: a qualitative interview study. Systematic reviews 2013; 2: 31.

Woods D, Trewheellar K. Medline and Embase complement each other in literature searches. BMJ 1998; 316: 1166.

Wright K, McDaid C. Reporting of article retractions in bibliographic databases and online journals. Journal of the Medical Library Association 2011; 99: 164-167.

Wu XY, Tang JL, Mao C, Yuan JQ, Qin Y, Chung VC. Systematic reviews and meta-analyses of traditional Chinese medicine must search Chinese databases to reduce language bias. Evidence-Based Complementary and Alternative Medicine 2013; 2013: 812179.

Yale Law School. GHJP Closes Two-Year FOIA Case Against Drug Manufacturer 2017. https://law.yale.edu/yls-today/news/ghjp-closes-two-year-foia-case-against-drug-manufacturer.

Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association 2009; 97: 21-29.

Young T, Hopewell S. Methods for obtaining unpublished data. Cochrane Database of Systematic Reviews 2011; 11: MR000027.

Zarin DA, Fain KM, Dobbins HD, Tse T, Williams RJ. 10-Year update on study results submitted to ClinicalTrials.gov. New England Journal of Medicine 2019; 381: 1966-1974.

Zeraatkar D, Pitre T, Leung G, Cusano E, Agarwal A, Khalid F, Escamilla Z, Cooper MA, Ghadimi M, Wang Y, Verdugo-Paiva F, Rada G, Kum E, Qasim A, Bartoszko JJ, Siemieniuk RAC, Patel C, Guyatt G, Brignardello-Petersen R. Consistency of covid-19 trial preprints with published reports and impact for decision making: retrospective review. BMJ Medicine 2022; 1: e000309.

Zhang B. Prompt Engineers or Librarians? An Exploration. Medical Reference Services Quarterly 2023; 42: 381-386.

Zwakman M, Verberne LM, Kars MC, Hooft L, van Delden JJM, Spijker R. Introducing PALETTE: an iterative method for conducting a literature search for a review in palliative care. BMC Palliative Care 2018; 17: 82.