75 - Telling the story of implementation: 10 years of collecting and using fidelity data

Abstract Information: If we want to study an intervention's outcomes, we should understand the degree to which it is being implemented. This poster will share the journey of how an evidence-based student support intervention has measured and used fidelity of implementation data for the last 10 years, from constructing a fidelity monitoring system, to assuring the system’s reliability and validity, to using fidelity results to inform coaching and strategic planning. This intervention, City Connects, is a systemic approach to helping students thrive by connecting each and every child to a tailored set of prevention, intervention, and enrichment services in the school and community (City Connects, 2022). In alignment with the conference’s focus on uplifting youth from historically underserved communities, City Connects is designed to promote support and equity among students with significant academic and financial needs (City Connects, 2022). More than 75% of City Connects students are students of color, more than 23% are students with disabilities, and about 25% are English language learners. The intervention has a substantive body of evidence demonstrating positive impacts for students including reduced absenteeism, higher achievement, and enhanced post-secondary outcomes (City Connects, 2022). To help sustain the impact of the program as the network expands, City Connects invested in a fidelity monitoring system (FMS) to evaluate whether the activities that lead to these outcomes for students are implemented as intended. This poster will focus on what City Connects has learned from its FMS over the past decade, along with why and how evaluators might incorporate fidelity monitoring into their work. The poster will describe the process used to develop the FMS and how its results are used by practitioners. The FMS development was rooted in implementation science literature and involved close collaboration between evaluators, the creators of the practice, and practitioners. Since its inception, the results of the FMS are annually disseminated to practitioners supporting schools and analyzed at the network level to identify areas for further development. This is in alignment with the conference theme of empowering practitioners through data to make decisions and become active participants in fidelity work. Furthermore, the poster will describe steps that City Connects has taken to examine the reliability and construct validity of its FMS. It provides evidence of the FMS’ measurement capabilities using Cronbach’s alpha analysis and principal component analysis. It also establishes the value of examining the reliability and validity of fidelity tools in evaluation. Lastly, this poster describes how City Connects is exploring the use of fidelity scores to help inform decision-making and strategic planning for the intervention. To help make program fidelity scores more interpretable, fidelity levels were created to facilitate a shared understanding of what constitutes “high”, “moderate”, or “low” fidelity. Furthermore, a scale anchoring approach, typically used in standardized assessment, was applied in an innovative way to create descriptions of each fidelity level and give stakeholders a sense of the types of activities that schools are likely to be doing consistently at different levels of fidelity.