136 - The Power of Formative Data to Yield Responsive Programmatic Improvement: Leveraging Student and Program Staff Voice in Evaluation

Abstract Information: Prioritizing diverse perspectives in evaluation uplifts new voices and results in meaningful change. In this session, we describe the importance of capturing student and partner voices and experiences through formative evaluation approaches, especially as they relate to program quality. We will engage the audience in interactive dialogue around the power of formative evaluation and offer actionable steps towards leveraging formative data to enhance evaluation and how to support a program in applying that data to yield improvements that are directly informed by participants and partners. Finally, we aim to promote the continued use of formative evaluation throughout the lifespan of a program. As a case study, we offer applied examples from an evaluation of FirstHand, an informal STEM education program. FirstHand is a program of the non-profit University City Science Center and delivers free, multi-week STEM programming to middle and high school students who attend some of Philadelphia’s most systematically under-resourced schools. Two years ago, FirstHand partnered with AnLar to conduct an impact study of the program through a 5-year randomized-controlled trial to understand how participating in Firsthand impacts students' beliefs, feelings of belonging in science, and achievement in science, in addition to formative data — which will be the focus of this session. Through this work, AnLar has collected two types of formative data (1) observations to assess program quality using the STEM Program Quality Assessment (STEM-PQA) to determine how and when the program is using practices known to facilitate positive experiences and youth development, and (2) semi-structured interviews with small groups of students and school staff to explore perceived impact across different school cultures. Together, these data have offered new perspectives that have yielded timely programmatic improvements and shaped decision-making at all levels of the program. At the program level, program observations and student and partner interviews have led to changes in daily programmatic activities such as curated daily reflection pages in student lab notebooks, and the addition of individual and class goal-setting to each session. At the staff level, formative data has provided self-directed professional learning opportunities for facilitators to review recordings of sessions they taught, and use those recordings to identify growth opportunities and goals based on the STEM-PQA rubric. At the organizational level, formative data has told a clear story of what students want to see and don’t want to see in the program as it scales. In this session, we will elaborate on what formative data was collected and how it was shared with and received by the program team and will highlight the action steps the program took in response to the data. Our goal is to open up a meaningful dialogue between program developers and evaluators to explore how and what formative data collection opportunities can lead to continued program discovery and innovation well into the lifespan of all programs.