Nelson Systematic Reviews to Answer Healthcare, Questions, 2e

72

Chapter 6 • Selecting Studies for Inclusion

Knowledge of the first reviewer’s decisions may bias the second reviewer. Also, this approach may result in reduced specificity of the retrieval process while increasing the number of full-text articles requiring review. Masking/Blinding of Study Selection Historically, there has been speculation that the selection of studies may be influenced by sys tematic reviewers’ knowledge of the study author’s identity or institution, journal, or year of publication. 19 However, masking or blinding reviewers to authors, institution, journals, and treatment groups did not have a clinically or statistically significant impact on the results of five meta-analyses in a study evaluating this effect. 20 The questionable benefit of masking reviewers to information about study sources generally does not warrant the substantial time and effort required to do so, 10–12 estimated at approximately 1.3 hours per paper. 20 Training and Pilot Testing Systematic reviewer training and pilot testing of the eligibility criteria are important in maxi mizing the reliability of the study selection process. Systematic review standards refer to this process and advocate the use of written documentation and testing to improve accuracy and consistency. 1 Reviewer training and pilot testing of eligibility criteria can begin with a small sample of studies (eg, 10% to 20%). Calculation of κ -statistics to measure reviewer agreement may be particularly useful in identifying problems during pilot testing. This process can gener ate discussions about the sources of variation in interpreting the review’s eligibility criteria that can lead to their refinement. However, it is a less useful gauge of the overall study selection pro cess because the main goal of dual review is to accurately identify all available relevant studies, not necessarily to achieve perfect agreement. ■ ■ DOCUMENTATION AND REPORTING Transparency is essential to the study selection process and requires careful documentation of the decisions about the eligibility of each article identified by the literature search. Disposition of each study should be documented, along with reasons for exclusion if appropriate. 1 Proper documentation and reporting of the study selection process allow for replication and indepen dent assessment of potential bias in study selection by systematic review users. Although thorough documentation of the study selection process is important, it requires substantial effort. Documentation includes recording separate inclusion decisions from mul tiple independent reviewers for both the abstract and full-text levels of the selection process and identifying discrepancies from dual review for further discussion and resolution. For more complex, technical, and large reviews involving numerous reviewers, reference management software, such as EndNote, ProCite, or Reference Manager, can be used to document inclusion in addition to their usual roles in formatting the bibliography. A standardized coding system for categorizing the inclusion decisions and reasons for exclu sion promotes adherence to the eligibility criteria, increases efficiency in reviewer assessment, and assists with organization and documentation. For example, a typical coding system could use the word “include” for each included study and a prespecified scheme of numbers or let ters to classify reasons for exclusions that mirror the PICOTS elements (ie, ineligible popula tion, intervention, or comparator). This system can be created by the systematic reviewers and adapted for specific research questions (Table 6.6). Alternatively, purpose-built software, such as DistillerSR, EPPI-Reviewer 4, RevMan, and oth ers, is available to assist systematic reviewers with this and other review processes. However, software for systematic reviews requires significant time in training and setup for each review (see Chapter 14). Its value to a review would be based on the strengths and limitations of spe cific software, experience of the team with systematic reviews as well as the software itself, and

Copyright © 2024 Wolters Kluwer, Inc. Unauthorized reproduction of the content is prohibited.