Nelson Systematic Reviews to Answer Healthcare, Questions, 2e

66

Chapter 6 • Selecting Studies for Inclusion

■ ■ SOURCES OF VARIATION IN APPLYING INCLUSION AND EXCLUSION CRITERIA

Chapter 4 describes considerations for developing inclusion and exclusion criteria for studies, such as how criteria can narrow or broaden the scope of the review and how to use experts to refine criteria to optimize clinical relevance. Although the eligibility criteria are designed to define the specific boundaries of the research questions in terms of the PICOTS elements, they usually cannot encompass all possible sources of variation. When systematic reviewers encoun ter unanticipated sources of ambiguity in the eligibility criteria, the potential for subjectivity and bias in the study selection process increases. Both inter- and intra-reviewer variation can occur. Systematic reviewers may differ in their interpretations of eligibility criteria, leading to vari ation in judgments about the relevancy of studies (inter-reviewer variation). The κ -statistic is a measure of the level of agreement between reviewers. Although inter-reviewer variation has been widely acknowledged, little research has been done to quantify its effect. 2–5 A prospective study evaluated variation between two independent systematic review groups using the same research questions and eligibility criteria. 5 When the numbers of overall included studies were considered, the groups had only fair agreement ( κ = 0.55). However, when the numbers of important included studies were compared, agreement was excellent ( κ = 0.88). Reasons for variation included differences in interpretations of the relevance of the oldest papers, multiple publications using the same or similar data, inclusion of publications of case series studies, and availability of data in publications. Additional inconsistencies reported in other evaluations include use of non-English language publications and nonrandomized studies, and different definitions of outcomes. 2–4 Variability is also affected by application of the best evidence approach during study selec tion. A core principle of systematic reviews is that they are based on the highest levels of evi dence available, and reviewers select and prioritize the highest quality and most applicable studies to address their research questions. 6 This approach generally excludes or minimizes inclusion of lower-level studies if enough higher-level studies are available. However, limita tions in the evidence are often first discovered during the study selection process. Variability is introduced when reviewers make different decisions about when and how to lower the evidence threshold, such as including studies that do not meet the prespecified criteria. Eligibility criteria may need to be refined to reduce this type of variability (Box 6.1). BOX 6.1 Modifying Eligibility Criteria during the Study Selection Process For a systematic review of the effectiveness of patient navigation to increase cancer screen ing in populations adversely affected by health disparities, systematic reviewers worked with a technical expert panel to develop and refine the inclusion criteria (Table 6.2). 7 TABLE 6.2 INCLUSION CRITERIA Population Patients from populations adversely affected by health care disparities in the United States (ie, racial and ethnic minorities, socioeconomically disadvan taged, underserved rural populations, sexual and gender minorities, others subject to discrimination) Intervention Patient navigation services to increase routine colorectal, breast, and cervical cancer screening Comparator Usual or alternate care for colorectal, breast, and cervical cancer screening While reviewing and selecting studies for inclusion, the systematic reviewers identified recurring methodological questions regarding whether a study should be included. For example, whereas one study may describe the intervention as “patient navigation” that

Copyright © 2024 Wolters Kluwer, Inc. Unauthorized reproduction of the content is prohibited.