61 - Advancing the Conversation: Uplifting Practitioner Voices in Evaluating Programs at Scale

Abstract Information: Through our firsthand experiences evaluating informal STEM scale-up programs, we have heard directly from local program staff that the individuals and organizations closest to youth and family participants – the community organization staff and educators – are commonly excluded from conversations regarding whether scaled programs should be done in their communities and if so, how. We have seen developer-based or funder-based assumptions during both the design phase and scale-up that are often grounded in deficit-based models of thinking about local communities, which can lead to pervasive challenges at scale. Even if collaboration is a priority during the design phase, common assumptions when programs scale persist, including: presumption of a local need or desire for the program beyond the initial context, objectivity of the STEM content and dominant ways of understanding the world, available staff time to implement the program, the organization’s physical capacity to implement the program, and staff comfort with STEM topics or leading informal STEM learning programs.

These assumptions lead to issues such as a lack of local staff buy-in, challenges with youth recruitment or attendance, struggles with sustainability beyond the life of the grant, and a lack of culturally relevant practices. Additionally, outcomes identified by the developer, funder, or evaluator may not be important to communities, while the selection of content, materials, or activities may reinforce stereotypes about communities, exacerbate historical traumas, or unintentionally devalue community identities or ways of understanding the world.

Beyond documenting these issues, what is the role of the evaluator in uplifting voices in the scaling process? What would it mean if as evaluators we embraced and planned for flexibility over fidelity in scaled programs? What would it mean for us if local sites defined their own program goals and outcomes instead of adhering to the overarching initial program goals? What if we focused on documenting the collaborations and trust building between communities and science experts in addition to outcomes such as the number of youth who intend to pursue STEM careers? What if we had teams that valued sharing failures and pivots as much as sharing evidence of success?

We invite you to this session to discuss whose voices matter in scale-up programs and how evaluators play into that. We will discuss how we have sought to elevate and honor the experiences and feedback of program implementers and their youth and families in informing our evaluation approaches. We will wrestle with our conceptions of scale and what it means for the design of our evaluations. The goal of this Roundtable session is to challenge past ideas and approaches to evaluating informal scale-up programs and come away with ideas to apply to a new approach to evaluating scale-up programs.