Nelson Systematic Reviews to Answer Healthcare, Questions, 2e

69

Chapter 6 • Selecting Studies for Inclusion

be made during the course of the review, but often are not anticipated in advance. Rules created to address these issues must be documented in the report and the revised protocol. Decisions to include non-English language publications and study abstracts or confer ence proceedings also contribute to variability across systematic reviews on the same topic. 2–4 Although these may be important issues in determining the validity of a given review, they generally do not contribute to variability in eligibility decisions within a review. Decisions made about the selection of studies based on these characteristics should also be clearly documented. ■ ■ PROCESS FOR STUDY SELECTION The goal of the study selection process is to ensure that the final body of evidence encompasses all available studies relevant to the research questions of the systematic review. The way to accomplish this is to consistently apply the prespecified eligibility criteria for including and excluding studies. However, even when the eligibility criteria are prespecified and detailed, a certain degree of subjectivity in their interpretation is unavoidable. For this reason, established standards for screening and selecting studies for a systematic review can be used to protect against reviewer bias and random error. 1,10–12 These include using dual review, conducting the assessments in two stages, and training reviewers. As with all steps in the systematic review process, the approach used for study selection should be prespecified and explicitly described in the methods sections of both the protocol and the review report. Dual Review Dual review refers to the process of involving at least two systematic reviewers in decisions about the selection of individual studies. This approach varies from review to review, rang ing from using the second reviewer to check the decisions of the first reviewer, to the second reviewer undertaking a completely independent assessment of all studies. Using two or more reviewers to independently screen and select studies is recommended, 1 and it is widely accepted that the reliability of the study selection process is maximized when multiple reviewers inde pendently assess potentially relevant studies.

The benefits of dual, independent review were demonstrated in a study specifically designed to estimate the accuracy and reliability of multiple reviewers when assessing the eligibility of approximately 11,286 randomized controlled trials (RCTs) of methods to improve postal ques tionnaires. 16 This study found that single reviewers missed an average of 8% of eligible reports (range, 0% to 24%) and adding a second, independent reviewer increased the number of RCTs correctly identified by an average of 9% (range, 0% to 32%). 16 Although this study demonstrated an advantage of using two independent reviewers, it did so under relatively narrow circumstances and may have actually underestimated the broader benefits of this approach. This study was limited to the selection of RCTs that may provide more explicit descriptions of their PICOTS characteristics 17 than observational studies. Use of two reviewers may be even more critical in reviews involving a wider variety of study designs that are traditionally more difficult to interpret. Also, among the four reviewers evaluated in this study, 16 three had previous experience with sys tematic review methods and no information was provided about their expertise in the relevant clini cal areas. As described in Chapter 3, in building the systematic review team, it is recommended that investigators and staff offer a broad range of expertise in the relevant clinical areas and in systematic review methods. It is reasonable to expect that wider variation in background and experience level may lead to wider variation in the probability of reliable decisions about study eligibility than was observed in the study. Additionally, previous research has shown that reviewers with relevant clinical content expertise may have strong viewpoints that can influence their interpretation and application of the eligibility criteria. 10 Therefore, as may be the case for assessing more complex study designs, use of two independent reviewers for study selection may also be important when the review team includes reviewers with different types of expertise (Box 6.2). Copyright © 2024 Wolters Kluwer, Inc. Unauthorized reproduction of the content is prohibited.