97 - Including New Voices in Evaluation: Addressing Challenges from Recruitment and Design

Abstract Information: Evaluators often must balance the need for a comparable group of participants with ensuring that participants are as diverse as possible to support generalizability. However, recruitment difficulties may result in threats to both study design and generalizability. While recruitment has generally been more challenging post-pandemic, evaluating a history curriculum adds another layer of difficulty given the ongoing debates around social studies content and standards at the national, state, and local levels. In this Birds of a Feather session, we will use our experience with recruitment for an initial efficacy study of the online Mission US history learning games to facilitate discussion about how recruitment challenges can both positively and negatively affect study design, participant diversity, and the interpretation of findings, and how to best address these challenges in ways that ensure evaluations include as diverse a group of participants as possible while still allowing for meaningful conclusions. This Birds of a Feather session will provide an opportunity to discuss recruitment challenges and solutions, and the tension between diversity and measuring impact. We will discuss how we identified challenges to recruitment, implemented creative solutions, and adjusted our study design. We will ask participants to think about how this kind of direct, national sampling affects the diversity of our sample, as well as the conclusions that can be reached about intervention impact. Participants will be able to deeply engage with questions, such as: "Which eligibility requirements are essential?"; "How can we best address the tension between including a wide range of classrooms and detecting an impact?"; and “In what ways can and should we adjust initial efficacy study designs as eligibility criteria shift?”