Miranda Cumpston and Jacqueline Chandler
Key Points:
- As new studies are completed, the results of reviews may become out of date and thereby provide misleading information to decision makers.
- Cochrane Reviews should be assessed periodically to determine whether an update is needed. The decision to update should be based on the continuing importance of the review question to decision makers and the availability of new data or new methods that would have a meaningful impact on the review findings.
- A review update provides an opportunity for the scope, eligibility criteria and methods used in the review to be revised.
- An update should be conducted according to the standards required for any review, with some additional requirements to ensure that any changes are managed appropriately and reported clearly to readers.
This chapter should be cited as: Cumpston M, Chandler J. Chapter IV: Updating a review. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). Cochrane, 2020. Available from www.training.cochrane.org/handbook.
IV.1 Introduction
Since its inception, Cochrane has sought to maintain its reviews to ensure they are updated to include the most recent evidence. Reviews that are out of date and do not incorporate all the available evidence risk providing misleading information to decision makers and other stakeholders.
Garner and colleagues define an update as “a new edition of a published systematic review with changes that can include new data, new methods, or new analyses to the previous edition” (Garner et al 2016). Adding new studies and new data can substantively change the findings of the review. Even where the new studies observe results consistent with the existing data, increasing the number of studies can improve precision of effect estimates, demonstrate wider applicability of the effect, or enable additional comparisons or subgroup analyses to be performed. The introduction of new review methods, such as updated risk of bias assessment tools or improved statistical analysis methods, can also change both the results and the certainty of the review’s findings. Examples of the impact of incorporating new information and methods are illustrated in Box IV.1.a.
All Cochrane Reviews should be assessed periodically to determine whether an update is needed. Some areas of research evolve rapidly, whereas others are more stable, and some research questions stop being relevant to decision makers. A report assessing 100 systematic reviews published between 1995 and 2005 concluded the median time to require an update was 5.5 years, although 23% of reviews were out of date within two years, 15% within one year, and 7% were already out of date at the time of publication (Shojania et al 2007). Authors of Cochrane Reviews should therefore consider both whether an update is warranted, and when it will be most beneficial for each specific review (see Section IV.2).
In some areas, authors are establishing ‘living’ systematic reviews that adopt a continual updating process, such as monthly searching followed by rapid incorporation of new evidence into the published review. Living systematic reviews are most likely to be appropriate for questions that are of high importance to decision makers, and for which new evidence is likely to be frequently published and to have an important impact on the review’s findings (Elliott et al 2017). Considerable resources are required to support such an ongoing process. Further discussion of living systematic reviews is presented in Chapter 22, Section 22.2.3.
Cochrane’s Methodological Expectations of Cochrane Intervention Reviews (MECIR), which guide the conduct and reporting of Cochrane Reviews, include expectations for updating reviews. See the online MECIR Manual for the 18 expectations specifically relevant to updates, although updated reviews should also meet the expectations that apply to all reviews. This chapter elaborates on those recommendations for the planning, conduct and reporting of Cochrane Review updates.
Box IV.1.a Examples of what factors might change in an updated systematic review (Garner et al 2016). Reproduced from Garner P, Hopewell S, Chandler J, MacLehose H, Akl EA, Beyene J, et al. When and how to update systematic reviews: consensus and checklist. BMJ 2016; 354: i3507 licensed under CC BY 3.0.
|
The decision to undertake an update of a review requires consideration of a number of different factors. Garner and colleagues conducted an international consensus process to establish good practice guidance for determining when a systematic review should be updated (Garner et al 2016). Their published framework and checklist can assist authors in thinking through these issues in a structured way (see Figure IV.2.a).
Figure IV.2.a Decision framework to assess systematic reviews for updating, with standard terms to report such decisions (Garner et al 2016). Reproduced from Garner P, Hopewell S, Chandler J, MacLehose H, Akl EA, Beyene J, et al. When and how to update systematic reviews: consensus and checklist. BMJ 2016; 354: i3507 licensed under CC BY 3.0.
When deciding whether to update a particular review, the first consideration should be to determine whether the review question remains relevant to decision makers, and is well-targeted to answer current questions in policy and practice. Knowledge of the particular field will be required to answer this question. Checking whether the existing review is frequently accessed or cited can also be useful to indicate whether there is a need to update. A second aspect to this question is whether the original review was conducted well and used appropriate methods (Garner et al 2016). If the review question remains fundamentally of interest, additions and improvements may be possible to enhance the review’s methods (see Section IV.3.4). Depending on the changes required, it may be more appropriate to conduct a new review from scratch meeting current standards. A comparison between currently recommended methods and the methods used in the review can identify any important changes required.
If the review remains important and is of a sufficient standard, then the next step is to consider whether there are any new studies, newly available information, or newly recommended methods that could be incorporated into the review. The existing version of the review may include details of ongoing studies identified at the time of its publication, for example through searches of trials registers, and these trials may now be complete. Some authors may choose to monitor the literature continually for new studies (e.g. through automated alerts), or may conduct a rapid scoping search for this purpose.
If either new information or new methodology is available, a critical next step is to evaluate whether incorporating these into the review would be likely to impact on its findings (Garner et al 2016). In some cases, this decision can be very straightforward, for example when the existing reviews findings are considered very uncertain (for example, using the GRADE approach to assessment, see Chapter 14). For some reviews, the findings are of very high certainty, and it is unlikely that new information will meaningfully impact the conclusions. In some cases, maintaining credibility through the incorporation of additional information and new methods is sufficient in itself to warrant updating (Garner et al 2016).
In some cases, although the main findings of the review may be unaffected, additional information may shed light on more nuanced effects of different variations on the intervention, different settings, additional outcomes, or population subgroups. In other cases, it may not be clear whether the extent of new information available will be enough to impact meaningfully on the results (Garner et al 2016).
To date there is no consensus on when to update a review (Tsertsvadze et al 2011), although several methods have been proposed (e.g. Sampson et al (2008), Shekelle et al (2011), Tovey et al (2011), Ahmadzai et al (2013), Takwoingi et al (2013)). These methods use signals to indicate the need for an update and the likely impact of new studies on existing conclusions. They include surveillance searches, contact with experts, and quantitative or qualitative assessments, or both. Chapter 22, Section 22.2, outlines a range of methods for surveillance of the literature and the interpretation of signals for updating, including statistical methods based on sample size calculations or the application of prediction equations to assess the impact of new evidence. Garner and colleagues also summarize a series of available methods (Garner et al 2016). Ultimately, review authors should make a judgement based on an individual assessment and their knowledge of the field covered by the review.
Published Cochrane Reviews are classified using an update classification system that draws on the system described by Garner and colleagues. This identifies whether the review is up to date, an update is planned, or no update is planned. A rationale is given if the status is that no update is planned, for example because the intervention has been superseded, the research area is neither active nor important, no new or ongoing studies have been identified, or new studies are unlikely to change the conclusions.
IV.3 Planning an update
Before embarking on an updated review, it is important to take the time to plan the process. Any proposed modifications or additions to the existing review should be planned in detail, and on occasion may require drafting a new protocol for the review. In addition, there are several issues unique to updates that should be considered.
Many of the approaches using new technologies designed to facilitate the review process are intended to support easier and more frequent updates. Further information is available in Chapter 4, Section 4.6.6, and Chapter 22, Section 22.2.4.
See the online MECIR Manual for expectations relevant to planning an update.
IV.3.1 Reconsidering review questions and eligibility criteria
Even within an overall question that has been agreed to remain relevant, an update is an opportunity to consider changes to the question and its scope. Authors should reconsider all elements of the review question (PICO), the eligibility criteria, comparisons and outcomes of interest. For example, evolving understanding of the problem may lead to the inclusion of a new comparison, an additional category of patients (e.g. children in addition to adults) or an important new outcome (e.g. adverse effects) that may not have been adequately addressed in the original review. Review authors may also wish to include additional objectives, such as addressing the economic aspects of the intervention or its implementation. Additional engagement with stakeholders may reveal current issues around which there is uncertainty (see Chapter 2).
Irrespective of whether the review question(s) change, there may be reason to amend the eligibility criteria for the review (see Chapter 3). For example, if a review includes both randomized trials and non-randomized studies and the former provide sufficient evidence to answer the review questions, it may be reasonable to decide to exclude non-randomized studies from subsequent updates of the review. Conversely, it may be reasonable to add non-randomized studies to a review that was previously restricted to randomized trials, to widen the evidence base, making use of methodological developments in critical evaluation of the validity of non-randomized studies (see Chapter 24).
IV.3.2 Splitting and merging reviews
As the body of evidence accumulates over time, a review may become too large for authors to manage (some of the largest Cochrane Reviews include hundreds of studies across multiple comparisons). It is sometimes appropriate to consider splitting the review into two or more reviews with more narrowly defined questions. For example, an early Cochrane Review investigated all interventions for shoulder pain. As this review became large and unwieldy over time, it was split into multiple separate reviews, each looking at an intervention category. One of these reviews looked at physiotherapy interventions for shoulder pain (Green et al 2003). As time went on, this review also became too large to manage, and was split into a number of reviews examining different physiotherapy interventions and specific types of shoulder pain (e.g. Page et al (2014a), Page et al (2014b), Page et al (2016a), Page et al (2016b)).
Narrower reviews may allow deeper investigation of specific intervention types, and more focused information for stakeholders, and may distribute the updating burden between several review author teams. On the other hand, narrower reviews can sometimes prevent readers from considering findings across all the interventions relevant to a decision (see Chapter 2, Section 2.3). Overviews of Reviews are an alternative option, allowing authors to summarize several more narrowly defined reviews that may have been split from a larger review (see Chapter V).
It is also possible for one or more narrower Cochrane Reviews to be merged into a larger review, where agreed by all authors that this would present a more useful synthesis for decision makers. For example, it might be concluded that a network meta-analysis to compare multiple intervention options for a particular condition would be more useful than an existing series of separate reviews of specific interventions (see Chapter 11).
IV.3.3 Planning the search strategy for an update
Once the scope and eligibility criteria for the update have been agreed, authors will prepare for an update by deciding on the appropriate search process and strategy.
A starting point for identifying new studies for inclusion may be those already identified as ongoing studies at the time of the existing version of the review. Following this, in some cases, the search strategy can be re-run as specified in the existing review, with the addition of date limits set to the period following the most recent search. However, an information specialist or healthcare librarian should be consulted to ensure the strategy remains appropriate. Changes to electronic databases, their access mechanisms and controlled vocabulary can require expert amendments to the search strategies. In addition, informed by the experience of the search for the original review, a decision may be made to modify the list of sources to be searched or search terms to be used (Garner et al 2016).
If important changes to the PICO for the review or the eligibility criteria have been made since the original search, or developments in the field have led to the emergence of new terms to be added to the search, it may be necessary to re-run parts of the search back to the earliest records, to ensure that any records relevant to new search terms were not missed in the original search.
IV.3.4 Planning the methods for an update
Methodological advances in systematic review conduct since publication of the original review may result in a need to revise or extend the methods of the review update (Shea et al 2006). Authors are encouraged to consult current guidance on review methods and compare these with the methods used in the existing review to identify important changes.
Examples of situations in which review methodology might be updated include:
- incorporating updated guidance on risk of bias assessment (see Chapter 7 and Chapter 8);
- using a new synthesis strategy, such as an improved method to perform a random-effects meta-analysis (see Chapter 10);
- incorporating GRADE assessments and ‘Summary of findings’ tables if not already included (see Chapter 14); and
- adopting new guidance on the structure and presentation of findings, such as structured tabulation of results in review using synthesis without meta-analysis (see Chapter 12).
Changes to the scope of the review, such as expansion to include different study designs or outcome data, will require planning for new methods appropriate to the data expected.
Where changes to the review methods are substantive, authors are encouraged to write a complete, updated protocol to guide the conduct of the review update. In some cases, it may be more appropriate to consider the work as a new review, rather than an update.
Specific methods developed for systematic reviews that conduct ongoing and prospective approaches to accumulating evidence to maintain review currency are outlined in Chapter 22. Formal sequential statistical methods that aim to address errors associated with repeating meta-analyses over time have been developed. However, such approaches are explicitly discouraged for updated meta-analyses in Cochrane Reviews, except in the context of a prospectively planned series of primary research studies (see Chapter 22, Section 22.4).
IV.3.5 Incorporating feedback and comments
Updating a published review provides an opportunity to consider any feedback or comments submitted to Cochrane or directly to the authors. Review authors are expected to be responsive to comments on their reviews, in the spirit of the scientific process and publication ethics. Comments may represent valid concerns and can usefully identify additional studies that were overlooked by the review authors.
IV.4 Conducting an update
An update of a review should be conducted according to the protocol, as closely as possible to the methods of the existing review while incorporating any planned changes (see Section IV.3). All steps should be conducted in accordance with the guidance presented throughout this Handbook.
A systematic search should be conducted for new studies (see Chapter 4), and the date of the search should be within 12 months of publication of the update. If new, potentially relevant studies are found, they should be assessed for inclusion in the review according to the eligibility criteria. If the existing review included records of any ongoing studies that are now complete, or studies for which classification as included or excluded was pending, newly available information should be sought and, where possible, final inclusion decisions made.
If new studies are to be included in the updated review, data should be collected (see Chapter 5) and risk of bias assessments completed for all new studies (see Chapter 7). On a practical note, when changes have been made to the scope or PICO of the review, tools such as the original data collection forms may need to be altered or extended and piloted again to ensure they are fit for purpose. This may also be needed if new software tools are to be used for data collection, or if a new author team has taken on the review, although existing templates and forms may be available from the original review authors or repositories such as the Systematic Review Data Repository (https://srdr.ahrq.gov/).
The findings of any new studies should be integrated into the synthesis of the review (see Chapter 10, Chapter 11, and Chapter 12), and GRADE assessments completed (or revised), taking full account of the new body of evidence (see Chapter 14).
If no new studies are found to be included in the review, authors should complete and publish the updated review (see Section IV.5). While not modifying the findings, including the details of an updated search will reassure readers and decision makers of the currency of the review.
See the online MECIR Manual for expectations relevant to conducting an update.
IV.4.1 Updating data from previously included studies
Since the time of publication, additional information may be available about one or more studies included in the existing review. For example, additional outcome data measured at later time points may now be available, or the study may have been corrected or retracted due to errors, fraud or a range of other reasons. It is important to search online journals or databases such as MEDLINE (if the study is indexed there) for any notifications, corrections or retractions.
Any additions or corrections should be incorporated into the information contained in the review, if relevant. The reasons for any retracted studies should be considered. In addition to the publication record, this information may be available in reports of investigations, such as by the authors’ institutions or funders. In those cases where data have been fabricated, they should be removed from the review analysis and a record made. Other studies by the same author(s) which would also be eligible for inclusion should be checked for similar issues, and a decision made as to whether they should similarly be removed. Further guidance on identifying corrected or retracted studies is provided in Chapter 4, section 4.4.6.
If a new comparison or a new outcome has been added to the review, it may be necessary to go back to the original included studies and check whether they included any information not previously collected that would be relevant to the update.
IV.5 Reporting an updated review
An updated review should meet the same standards of reporting as any review (see Chapter III), while ensuring that all updated information and changes made to the scope and methods of the review are reported clearly. The details of any changes, including justifications for the decisions made, can be documented in the ‘Differences between review and protocol’ section of the review. Authors should clearly alert readers that this is an update of an earlier version, including statements in the Abstract and Background sections of the review.
Appearing at the beginning of the review, the Background section is not directly impacted by an update, but authors may wish to review the content of the Background to ensure that it remains fit for purpose. Discussions of the prevalence or incidence of a condition, new insights into the mechanism of action or impact on populations, or descriptions of current practice options may be updated. Any references to time, such as words like ‘recently’ or ‘in the next five years’, should be amended or, if possible, removed.
Reporting the details of the updated search alongside the search information in the existing review can become quite complex, especially if there have been several updates to the review over time. There are several approaches to reporting the results of an updated search:
1. An integrated approach describes all searches together, which may be most feasible if the same search was repeated.
2. An incremental approach adds information at each update to describe explicitly which searches were done for the update, retaining all information about previous searches.
3. A replacement approach describes only the searches done for the update, using the previous review as one source of studies.
4. A hybrid approach describes only the searches done for the update in the main text, using Appendices to provide information about previous searches.
The updated search should also be presented in a PRISMA-type flow diagram (see Chapter 4, Section 4.5). Again, there are options as to how to present the results of multiple searches coherently in the diagram. Authors can retain the results of previous searches in the review and supplement with information about studies identified in the update or, alternatively, present only information about searches in the current update, with the previous version of the review serving as one particular source of studies.
The methods and results described throughout the review and its summaries (including the ‘Summary of findings’ table, Abstract and Plain Language Summaries) should be checked to ensure they still reflect the methods used accurately. Where the review is considered a ‘living’ systematic review, and regular updates are planned, additional methods should be included to describe the timing and nature of this process (see Chapter 22, Section 22.2).
The extent of revision to the Results of the review will depend on the influence of the new data on the results of the review. Examples include:
- the addition of small studies bringing about no change in the results or conclusions of the review (and so requiring very little revision of the text);
- increased certainty of pre-existing results and conclusions (requiring some modification of the text); and
- a change in the conclusion of a review (requiring a major rewrite of the Results, Discussion, Conclusion, ‘Summary of findings’ table, Abstract and Plain Language Summary).
When reporting the results, it is more helpful to readers to present an integrated picture of the overall results, rather than sequential or separate results for the update (especially where there have been separate updates), although the impact of an update on the overall conclusions may be of interest to discuss when interpreting the results.
Finally, authors should check that nothing else in the review requires editing, such as references to other Cochrane Reviews that may have been updated, or additions to the Acknowledgements. The ‘Declarations of interest’ sections of the review should be updated.
See the online MECIR Manual for expectations relevant to reporting an updated review.
IV.5.1 Changes in authorship
If there is a change in the authorship of the review, such as new authors joining the team, or an entirely new team of authors updating the review, the by-line (list of authors) may need to be changed. The decision regarding who is named in the by-line of an updated review, and in what order, should be assessed in terms of contributions to content in the updated version of the review (which will include historical content), and responsibility for approving the final content of the manuscript. If an author is no longer actively contributing to or involved in the approval of an updated review, the author should not be listed in the by-line of the new version and should be named in the Acknowledgements section. The contributions of all authors to both the update and earlier versions of the review should be described in the ‘Contributions of authors’ section.
See Cochrane’s policy on authorship and contributorship for Cochrane Reviews for more information.
IV.6 Chapter information
Authors: Miranda Cumpston and Jacqueline Chandler
Acknowledgements: This chapter builds on earlier versions of the Handbook. Contributors to earlier versions include Julian Higgins, Rachel Marshall, Ruth Foxlee and members of the former Updating Working Group (Mike Clarke, Mark Davies, Davina Ghersi, Sally Green, Sonja Henderson, Harriet MacLehose, Jessie McGowan, David Moher, Rob Scholten (convenor) and Phil Wiffen). David Tovey, Carol Lefebvre and Sally Hopewell provided comments on earlier versions. Rachel Marshall re-drafted version 5.1 on which this version was based with input from Harriet MacLehose. Mona Nasser contributed to section IV.2.1. Rachel Churchill contributed to the re-structuring of this version. The work of Garner and colleagues (Garner et al 2016), a key reference used throughout, was based on a consensus meeting of experts funded by Cochrane.
IV.7 References
Adams SP, Tsang M, Wright JM. Lipid lowering efficacy of atorvastatin. Cochrane Database of Systematic Reviews 2012; 12: CD008226.
Ahmadzai N, Newberry SJ, Maglione MA, Tsertsvadze A, Ansari MT, Hempel S, Motala A, Tsouros S, Schneider Chafen JJ, Shanman R, Moher D, Shekelle PG. A surveillance system to assess the need for updating systematic reviews. Systematic Reviews 2013; 2: 104.
Elliott JH, Synnot A, Turner T, Simmonds M, Akl EA, McDonald S, Salanti G, Meerpohl J, MacLehose H, Hilton J, Tovey D, Shemilt I, Thomas J, Living Systematic Review N. Living systematic review: 1. Introduction-the why, what, when, and how. Journal of Clinical Epidemiology 2017; 91: 23-30.
Garner P, Hopewell S, Chandler J, MacLehose H, Schünemann HJ, Akl EA, Beyene J, Chang S, Churchill R, Dearness K, Guyatt G, Lefebvre C, Liles B, Marshall R, Martinez Garcia L, Mavergames C, Nasser M, Qaseem A, Sampson M, Soares-Weiser K, Takwoingi Y, Thabane L, Trivella M, Tugwell P, Welsh E, Wilson EC, Schünemann HJ, Panel for Updating Guidance for Systematic Reviews (PUGs). When and how to update systematic reviews: consensus and checklist. BMJ 2016; 354: i3507.
Green S, Buchbinder R, Hetrick S. Physiotherapy interventions for shoulder pain. Cochrane Database of Systematic Reviews 2003; 2: CD004258.
Higgins JPT. Convincing evidence from controlled and uncontrolled studies on the lipid-lowering effect of a statin. Cochrane Database of Systematic Reviews 2012: ED000049.
Page MJ, Green S, Kramer S, Johnston RV, McBain B, Buchbinder R. Electrotherapy modalities for adhesive capsulitis (frozen shoulder). Cochrane Database of Systematic Reviews 2014a; 10: CD011324.
Page MJ, Green S, Kramer S, Johnston RV, McBain B, Chau M, Buchbinder R. Manual therapy and exercise for adhesive capsulitis (frozen shoulder). Cochrane Database of Systematic Reviews 2014b; 8: CD011275.
Page MJ, Green S, McBain B, Surace SJ, Deitch J, Lyttle N, Mrocki MA, Buchbinder R. Manual therapy and exercise for rotator cuff disease. Cochrane Database of Systematic Reviews 2016a; 6: CD012224.
Page MJ, Green S, Mrocki MA, Surace SJ, Deitch J, McBain B, Lyttle N, Buchbinder R. Electrotherapy modalities for rotator cuff disease. Cochrane Database of Systematic Reviews 2016b; 6: CD012225.
Prasad K, Singh MB, Ryan H. Corticosteroids for managing tuberculous meningitis. Cochrane Database of Systematic Reviews 2016; 4: CD002244.
Sampson M, Shojania KG, McGowan J, Daniel R, Rader T, Iansavichene AE, Ji J, Ansari MT, Moher D. Surveillance search techniques identified the need to update systematic reviews. Journal of Clinical Epidemiology 2008; 61: 755-762.
Shea B, Boers M, Grimshaw JM, Hamel C, Bouter LM. Does updating improve the methodological and reporting quality of systematic reviews? BMC Medical Research Methodology 2006; 6: 27.
Shekelle P, Newberry S, Wu H, Suttorp M, Motala A, Lim Y, et al. Identifying Signals for Updating Systematic Reviews: A Comparison of Two Methods (Prepared by: The RAND Corporation, Southern California Evidence-based Practice Center, Santa Monica, CA under Contract No 290-2007-10062I; Tufts Evidence-based Practice Center, Tufts Medical Center, Boston, MA under Contract No 290-2007-10055I; University of Ottawa Evidence-based Practice Center, Ottawa, Canada under Contract No 290-2007-10059I). Rockville (MD): Agency for Healthcare Research and Quality; 2011.
Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Annals of Internal Medicine 2007; 147: 224-233.
Takwoingi Y, Hopewell S, Tovey D, Sutton AJ. A multicomponent decision tool for prioritising the updating of systematic reviews. BMJ 2013; 347: f7191.
Tovey D, Marshall R, Bazian Ltd, Hopewell S, Rader T. National Institute for Health Research Cochrane-National Health Service Engagement Award Scheme Fit for purpose: centralised updating support for high-priority Cochrane reviews 2011. http://www.editorial-unit.cochrane.org/fit-purpose-centralised-updating-support-high-priority-cochrane-reviews.
Tsertsvadze A, Maglione M, Chou R, Garritty C, Coleman C, Lux L, Bass E, Balshem H, Moher D. Updating comparative effectiveness reviews: Current efforts in AHRQ's Effective Health Care Program. Journal of Clinical Epidemiology 2011; 64: 1208-1215.
Zani B, Gathu M, Donegan S, Olliaro PL, Sinclair D. Dihydroartemisinin-piperaquine for treating uncomplicated Plasmodium falciparum malaria. Cochrane Database of Systematic Reviews 2014; 1: CD010927.