Building Structure and Flexibility: Lessons from the First Year of an Evaluation of Full-Service Community Schools in Kentucky

Abstract Information: Evaluations that take multiple years need both structure and flexibility (Fitzpatrick et al., 2010). It is helpful to have, in a long-term evaluation, a sustainable study design, documentation of key decisions, and processes and routines for regular and productive communication between the evaluation and program implementation teams. At the same time, successful evaluation teams can pivot as program priorities or implementation conditions change. Establishing this structure alongside flexibility is challenging—but the first year of a multi-year study can be an important time to establish the norms, practices, and systems that set up evaluation and program teams for long-term success (Weddle, 2023). In this session, we will draw on our experiences from the first of a five-year evaluation to share lessons that focus on ways in which we are establishing both structure and flexibility in our systems and processes as an evaluation team—as well as in the substantive focus of the evaluation. We are evaluating efforts to scale full-service community schools across Kentucky. Funded by the U.S. Department of Education, this multi-year evaluation includes a developmental study, supports to the program team for implementation, as well as an impact evaluation. The evaluation also draws on quantitative and qualitative data collection and analyses. Finally, we are a large evaluation team, housed at a university-based evaluation center, that includes faculty, research staff (at the pre- and post-doctoral level), and graduate students. For this reason, we anticipate turnover within the evaluation team, especially at the student-level. If accepted to the conference, in this session, over the course of three papers, we will tackle the substantive and logistical decisions we have made around key questions like: -- How do we balance structure and flexibility in measuring implementation and impact outcomes? -- To what extent might district- or school-specific contexts shape the focus of the evaluation study? -- In what ways can we invite local stakeholders into our evaluation work? -- How can we shape our work given the changing nature and skillset of our team? -- How do we navigate and build trust as external evaluators in the process? The first paper draws on Lincoln and Guba’s (1980) seminal piece to highlight the ways in which we are thinking about merit and worth in this particular project—in both, the short- and long-term. This distinction is important for us as we help our partners with implementation challenges and determine the parameters of our impact study. The second presentation will draw on Patton (2011) to focus on the challenges we face in supporting our program partners as developmental evaluators and those assessing program implementation fidelity. The final paper will draw on insights from the literature on research-practice partnerships (Penuel & Gallagher, 2017) to highlight the ways in which we are developing structure and flexibility into our qualitative evaluation work while also building rapport and trust with both program staff as well as district- and school-based staff working on full-service community schools.