Systematic Reviews to Answer Health Care Questions

186

Chapter 12 • Assessing and Rating the Strength of the Body of Evidence

TABLE 12.2 DOMAINS USED IN RATING STRENGTH OF EVIDENCE MEASURES DOMAIN DEFINITION CONSIDERATIONS GRADE 5,11 AHRQ EPC 3 Study limitations High, medium, low

The extent to which methodological defi ciencies across studies could bias results The relevance of the evidence to the research question based on patient population, interventions, compara tors, and outcomes

What are the quality (risk of bias) ratings of individual studies?

No limitations, serious limita tions, very seri ous limitations No indirectness, serious indi rectness, very serious indirect ness; includes applicability No inconsis tency, serious inconsistency, very serious inconsistency

Directness

Do most studies address the PICOTS elements of the key question?

Direct, indirect; applicability assessed sepa rately

Consistent, inconsistent, unknown/none

Consistency Degree of similarity in the direction and mag nitude of effect of dif ferent studies in a body of evidence

How consistent are results based on over lapping confidence intervals, similarity of point estimates, between-study vari ance, measures of het erogeneity (eg, I 2 ) as relevant to the review? Are the number of events sufficiently high and width of the confidence intervals adequate to support a meaningful effect? Is there evidence of small study effects or selective reporting or publishing?

Precision Degree of certainty

No imprecision, serious impreci sion, very seri ous imprecision

Precise, imprecise

surrounding an estimate of effect for a specific outcome

Reporting bias

Includes publication bias (the entire study is missing), outcome reporting bias (specific outcomes that are mea sured are not reported), and analysis reporting bias (specific preplanned analyses are conducted but not reported) The size of the effect is so large that results are believable despite potential study bias Effects are greater with increasingly higher levels of interventions or exposures Confounding factors lead to an underesti mate of the effect of an intervention

Publication bias assessed sepa rately as unde tected, strongly suspected; other biases considered under study limitations Level upgraded by one (if RR >2 or <0.5) or two categories (if RR >5 or <0.2) Level upgraded by one

Undetected, suspected

Large magnitude of effect (strength of association) Dose– response association

What is the magni tude of effect and width of the confi dence intervals?

Weak, strong; taken into account to improve rating Weak, strong; taken into account to improve rating Weak, strong; taken into account to improve rating

Copyright © 2024 Wolters Kluwer, Inc. Unauthorized reproduction of the content is prohibited.

Is a dose response effect present?

Opposing plausible

What are possible biases and confound ing factors that were not considered but could have influenced the estimate of effect?

Level upgraded by one

residual bias and confounding

Abbreviations: AHRQ EPC, Agency for Healthcare Research and Quality Evidence-based Practice Centers; GRADE, Grading of Recommendations Assessment, Development and Evaluation; RR, relative risk.