177 - From Participation to Empowerment: Engaging Stakeholders in Instrument Development

Abstract Information: Creating valid and reliable instruments to assess evaluation outcomes requires intentional and authentic engagement with the communities that they will be used to evaluate. ICF, an evaluator of the Arizona GEAR UP (AZGU) program, adopted a participatory utilization framework to develop a culturally responsive scale for assessing students’ perceptions of college-going culture within their school communities. We convened a Participatory Evaluation and Sustainability (PES) team consisting of students, teachers, GEAR UP coaches, and evaluators to identify relevant constructs to measure college-going culture. The PES team identified the following key constructs: student excitement, safe space, engagement outcomes, inequity, visual ques for college-going culture, relationships with peers, and student and staff perceptions of support. Guided by theoretical frameworks such as Community Cultural Wealth, The Nine Principals of College-Going Culture, and the program’s own 5E Model, the evaluation team developed questions and items based on the suggestions provided by the PES team. After reviewing the literature, these questions and items were then refined through a series of facilitated discussions with the PES team using Mural Online Collaboration to ensure they effectively operationalized stakeholder perceptions of college-going culture and the program model. This iterative process culminated in the development of a 23-items scale measured on a 4-point Likert scale, where higher scores indicate stronger agreement. The final scale was administered to a sample of 3438 students. The reliability of the scale has been demonstrated by a Cronbach’s alpha value=0.923 and evidence for a one-factor model with adequate model fit. Creating valid and reliable instruments is essential for effective evaluation and our approach demonstrates the power of inclusive, collaborative processes in achieving this goal.