Tuning Out the Chatter: Experiences Learning Which Voices to Amplify in Program Evaluation

Abstract Information: As evaluators, we bear a responsibility to both our clients/project teams and the study participants who trust us with their feedback. With each study, we make the choice on how to best represent the interests of our clients or project team to audiences. We also bear the responsibility of filtering through provided feedback from qualitative data collection and deciding how to best communicate the perspectives or experiences of project activity recipients. Often, the insights provided through interviews, focus groups, and open-ended survey questions align closely with project goals, which allow us to help our clients make data-driven decisions based on project impact, strengths, and opportunities for growth. On occasion, there are perspectives shared that – while valid – can be deeply misaligned with overarching program intent or guiding ethical practices; other times, power dynamics in group conversations prevent everyone from contributing, or sentiments shared touch on broader issues important for clients/project teams to know. These perspectives can sometimes challenge our own personal belief systems and biases and lead us questioning how to report findings and share feedback with clients in a way that ensures we are still amplifying the right voices. In this roundtable, Pacific Research and Evaluation will share recent experiences they’ve navigated determining how to appropriately and respectfully amplify the voices that most need to be heard. Examples of experiences that prompted our exploration of this topic are described below: • In an ongoing study focused on increasing feelings of belonging among first-time-in-college students in a predominantly minority-serving community, open-ended survey data from a small but vocal population of students from dominant demographic groups demonstrated feelings of being excluded or unwelcome on campus as a result of initiatives that promote diversity, equity, and inclusion (DEI). Some of these respondents believed these DEI initiatives to be illegal in their state and used the survey to threaten to tell government officials that programming should be shut down. • On occasion, we facilitate focus groups where one or two participants dominate the conversation and other participants provide very little input, if any. This can sometimes occur when different power dynamics are at play—despite our efforts to manage this in advance—or when our attempts to give others an opportunity to speak are ignored. While the data shared from dominating focus group participants may be relevant to the program being discussed, it leaves us with data that does not amplify all voices. This roundtable will allow us to share about some of these situations where we received data that could be classified as “chatter” and share what techniques have helped us steer through these circumstances. However, rather than coming to a consensus on how to navigate tough moments in evaluation activity, this roundtable will provide attendees with a space to work through needs and solutions, while building bonds with peers.