“Building Evaluation Capacity: The Importance of Proactive and Responsive Evaluation Redesign when Evaluating Complex and Large-scale Programs”

Abstract Information: The session discusses the importance of building evaluation capacity through a description of the challenges and lessons learned while evaluating a complex, large-scale program in the eighth-largest school district in the country. The presenters will describe the ways these challenges were addressed through the lenses of evaluation capacity building (ECB) (Baizerman, Compton, & Hueftle Stockdill, 2002), intervention complexity (Kainz, Metz, & Yazejian, 2021), and developmental evaluation (Patton, 2011). Early career and more seasoned evaluators will benefit from the description of the shared journey of learning, adaptation, and (ultimately) complexity reduction to meet the district’s requirement to demonstrate program impact. Participants will receive a handout (“checklist”) of considerations and recommendations for designing evaluations of complex initiatives that are already underway when evaluators are engaged. Newer evaluators will see a model for evaluation partnership that emphasizes collaboration with the funding organization at every step of the process from refining the evaluation questions through reporting.
The program that was the focus of the evaluation was funded through the Elementary and Secondary Education Emergency Fund (ESSER). It consisted of instructional activities implemented by over 800 intervention teachers who were deployed to assist struggling learners across the district in the wake of the COVID-19 pandemic. Challenges to the evaluation design and implementation included: 1) a lack of a coherent definition of the program across the district, 2) considerable variability in program implementation methods across schools and even within schools, 3) lack of student outcome data that could be clearly linked to students receiving the treatment, 4) high levels of attrition among staff who were hired to carry out the program, 5) use of program staff to perform duties outside of the intended scope of the program, and 6) changes in personnel among district leadership and evaluation personnel that altered initial program evaluation design.