Conducting a Large Systematic Review: A View from the Trenches

Abstract Information: High-quality systematic reviews and meta-analyses hinge on best practices researchers embrace while conducting them, whether in education, health, social sciences, etc. As the field of research synthesis has flourished, resources related to how to conduct systematic reviews, meta-analyses and best practices have grown exponentially. However, limited guidance exists for researchers conducting large-scale reviews. This study draws on a research team’s experiences and examines opportunities and challenges in conducting a large systematic review. Specifically, this study stems from experiences in conducting a systematic review and meta-analysis related to personalized adaptive learning in undergraduate mathematics. A total of 10,587 non-duplicated studies were identified at the abstract and title screening phase from a search conducted in 13 databases. This presentation will discuss lessons learned and best practices resources for conducting the literature search and screening abstracts/titles and will assist applied researchers interested in conducting large systematic reviews. What is a Large Systematic Review? There are no minimum requirements for what constitutes ‘too small’ for a systematic review, with even zero studies constituting importance. ‘Large’ systematic reviews have been defined as searches that result in 5,000 or more non-duplicated studies at the abstract/title stage (Polanin et al., 2019). Large systematic reviews have significant evidential implications for decision and policy-making for diverse fields (Brown et al., 2012, Polanin et al., 2019). What Does it Take to Manage a Large Systematic Review Without Compromising on Rigor? Understand Best Practices. Numerous ‘best practices’ exist but they are scattered widely across many publications. For example: Preferred Reporting Items for Systematic Reviews and Meta-Analyses, PRISMA (Moher et al., 2009) and the Cochrane Handbook (https://training.cochrane.org/handbook), in addition to many lesser-known methods (e.g., Polanin et al., 2019). Refine the Search Strategy. This is a multi-step process. Development and operationalization of the search strategy began with consideration of terms to identify the population, intervention, comparison, and outcomes (PICO) (Richardson et al., 1995). These were framed from the research question, as well as terms to identify the type of study design (i.e., randomized trial or quasi-experimental) (Higgins et al., 2022). This was supplemented by the project’s advisory board and searches of publisher websites with known personalized learning platforms/curricula to identify additional search terms. Consultations with librarians also helped identify terms. Consider Manageability. Thirteen databases were searched, and several of these indexed conference proceedings. Guidelines do not exist on how to manage eligibility screening for these sources which often contain hundreds of entries. Through the presentation, we will share the process we followed to manage screening conference proceedings. Select Software. No single software program meets all needs in a systematic review, from citation searching to abstract/title screening to data extraction to statistical analyses, and thus knowledge of auxiliary software is needed (Tantry et al., 2021). Conclusion As systematic reviews continue to grow as a method for evidence, coupled with ever increasing publications, researchers are likely to find themselves conducting a ‘large’ systematic review. This work complements the existing guidance in the field as it relates to addressing the logistics of conducting large systematic reviews.