Supplementary material 9 to: Focused review format prototype

Flemyng E, Mitchell D
https://doi.org/null

The material in this section has been supplied by the author(s) for publication under a Licence for Publication and the author(s) are solely responsible for the material. Cochrane has reviewed this material, but Cochrane has not copyedited, formatted or proofread. Cochrane accordingly gives no representations or warranties of any kind in relation to, and accepts no liability for any reliance on or use of, such material.

Back to top

GRADEing the certainty of the evidence

This supplementary material provides detailed information about downgrading or upgrading the certainty of the evidence during the implementation of GRADE.

Outcomes

Certainty of the evidence (GRADE)

Reasons for certainty GRADEing

Treatment failure: need for systemic corticosteroids (ITT)

46 weeks

MODERATE

Due to risk of bias

Studies carrying 13.3% of the analysis weight had an overall high risk of bias, and studies carrying a further 49.8% of the weight had some concerns. Studies with overall low risk of bias accounted for approximately a third of the weight of the analysis. Biases arose mostly in domains 2 and 3 (deviations from the intended interventions and missing data), often relating to assumptions that had to be made when there were differences between the way the study reported the outcome and how it was needed for the analysis, or uncertainty regarding the population used for the study analysis (−1 for risk of bias).
We did not prespecify bounds for downgrading for imprecision or concluding no difference between treatments. The upper and lower limits of the confidence interval may be considered clinically important benefit or harm of the intervention, but we did not consider it sufficient to downgrade given the number of events and participants in the analysis (no downgrade for imprecision).

Treatment failure: need for systemic corticosteroids (of those starting inhaler)

45 weeks

VERY LOW

Due to inconsistency, imprecision, and very serious risk of bias

All studies were well‐matched to our review question. We resolved uncertainties in the definitions of outcomes through contact with study authors. Where outcome definitions or the populations used for analysis (e.g. ITT or those taking the study inhaler) were unclear or differed from what was defined in the review protocol, this was accounted for as missing data and deviation from the intended intervention in the risk of bias assessment (no downgrade for indirectness across outcomes).
Upper and lower confidence intervals include important benefit of increased or stable ICS (−1 imprecision).
I2 = 42%, P = 0.11; clear variation noted between direction and magnitude of study results by visual inspection of the forest plot (−1 inconsistency).
Studies carrying 81.6% of the analysis weight had overall high risk of bias. Biases mostly arose in domains 2 and 3 (deviations from the intended interventions and missing data), relating to assumptions that had to be made to include imperfect data in the review analysis (e.g. where the population used for the study analysis was unclear or differed from the population defined for the review analysis). There was also risk of bias from unclear and inconsistent implementation of criteria for initiating the study inhaler in some studies (−2 for risk of bias).

Unscheduled physician visits

44 weeks

LOW

Due to very serious imprecision

All studies were well‐matched to our review question. We resolved uncertainties in the definitions of outcomes through contact with study authors. Where outcome definitions or the populations used for analysis (e.g. ITT or those taking the study inhaler) were unclear or differed from what was defined in the review protocol, this was accounted for as missing data and deviation from the intended intervention in the risk of bias assessment (no downgrade for indirectness across outcomes).

Several studies did not appear in the analysis, but contact with study authors meant this was unlikely due to selective reporting (no downgrade for publication bias).
Three studies observed 136 events leading to very wide confidence intervals, which made the result very difficult to interpret (−2 imprecision).
No studies in the analysis were at overall high risk of bias, although there were some concerns for a study carrying 56% of the weight (no downgrade for risk of bias).

Unscheduled acute care, ED visit, or hospital admission

47 weeks

VERY LOW

Due to risk of bias and very serious imprecision

All studies were well‐matched to our review question. We resolved uncertainties in the definitions of outcomes through contact with study authors. Where outcome definitions or the populations used for analysis (e.g. ITT or those taking the study inhaler) were unclear or differed from what was defined in the review protocol, this was accounted for as missing data and deviation from the intended intervention in the risk of bias assessment (no downgrade for indirectness across outcomes).

Only 12 events in the analysis, leading to substantial imprecision in the estimate. Two studies did not observe any events and so did not contribute to the effect estimate (−2 imprecision). A large amount of heterogeneity between the two contributing study effects warranted downgrading for heterogeneity (I2 = 62%), but was captured by imprecision and very low grading.

Serious adverse events

48 weeks

VERY LOW

Due to very serious risk of bias and imprecision

All studies were well‐matched to our review question. We resolved uncertainties in the definitions of outcomes through contact with study authors. Where outcome definitions or the populations used for analysis (e.g. ITT or those taking the study inhaler) were unclear or differed from what was defined in the review protocol, this was accounted for as missing data and deviation from the intended intervention in the risk of bias assessment (no downgrade for indirectness across outcomes).

Studies contributing the majority of the weight in both adverse events analyses were at overall high risk of bias, primarily in domains 2 and 3 (deviations from intended interventions and missing data), and additionally in domains 4 and 5 (measurement of the outcome and selection of the reported result) for non‐serious adverse events (−2 for risk of bias).
Confidence intervals included a significant increase in adverse events on increased‐dose ICS and did not exclude the possibility of no difference against stable ICS. Very few events were included in either of the adverse event analyses (−1 imprecision).

Non‐serious adverse events

43 weeks

VERY LOW

Due to very serious risk of bias and imprecision

All studies were well‐matched to our review question. We resolved uncertainties in the definitions of outcomes through contact with study authors. Where outcome definitions or the populations used for analysis (e.g. ITT or those taking the study inhaler) were unclear or differed from what was defined in the review protocol, this was accounted for as missing data and deviation from the intended intervention in the risk of bias assessment (no downgrade for indirectness across outcomes).

Studies contributing the majority of the weight in both adverse events analyses were at overall high risk of bias, primarily in domains 2 and 3 (deviations from intended interventions and missing data), and additionally in domains 4 and 5 (measurement of the outcome and selection of the reported result) for non‐serious adverse events (−2 for risk of bias).
Confidence intervals included a significant increase in adverse events on increased‐dose ICS and did not exclude the possibility of no difference against stable ICS. Very few events were included in either of the adverse event analyses (−1 imprecision).

Duration of exacerbation ‐ time to symptom recovery and lung function recovery

52 weeks

LOW

Due to risk of bias and imprecision

All studies were well‐matched to our review question. We resolved uncertainties in the definitions of outcomes through contact with study authors. Where outcome definitions or the populations used for analysis (e.g. ITT or those taking the study inhaler) were unclear or differed from what was defined in the review protocol, this was accounted for as missing data and deviation from the intended intervention in the risk of bias assessment (no downgrade for indirectness across outcomes).

Upper and lower confidence intervals include important benefit of increased or stable ICS (−1 imprecision).

Several studies did not appear in the analysis, but contact with study authors meant this was unlikely due to selective reporting (no downgrade for publication bias).