1053 - Impact: The Fundamentals No One Taught You (and Every Evaluator Needs to Know)

Abstract Information: What does “impact” really mean? You may be surprised. In this half-day workshop, participants will learn that impact has multiple definitions that fit into a single framework that connects logic models, systems thinking, and experiments. However, this is not a workshop on experimental design. It was developed to help evaluators, social investors, program designers, and managers reason critically about impact and how to measure it. The workshop is divided into three sections: (1) language traps and misconceptions, (2) the concept of impact, and (3) strategies for measuring impact. Participants will make light-hearted use of metaphors—time machines, the multi-verse, and all-seeing Zeus—to explore abstract concepts and apply them to practical purposes. Activities and games will reinforce what is learned while having fun. By the end, participants will be able to (1) use impact vocabulary correctly, (2) distinguish between the concept of impact and impact measurement, (3) recognize there are different conceptions of impact and organize them in a single framework, (4) judge the credibility of impact claims, and (5) make appropriate choices among the “big five” strategies for measuring impact. Participants will be provided with a workbook and access to free resources to continue their learning. All are welcome.

Relevance Statement: There is an acute need for a course exclusively on impact. This is one of the few (perhaps the only one) that offers working professionals and students an introduction to the topic. Impact is an essential concept in evaluation, yet many evaluators have an incomplete understanding of it. Most learn about impact in the context of logic models, research design, or reporting requirements. If so, their learning is incomplete and may promote misconceptions that affect the quality of their work. One reason for knowledge gaps and misconceptions is that impact has multiple valid definitions (in addition to many that are not valid). This has been noted by others (for example, White, 2009), but they only touch the surface of the problem. Even pioneers in the field of evaluation like Suchman (1968) were bamboozled by the term’s multiple meanings. He defines impact in different ways in the same piece of writing without apparently noticing. As the fields of social finance and social entrepreneurship continue to grow, the concept of impact is becoming more important. What sets professionals in these fields apart from others is their pursuit of impact. However, most have no formal training about impact, which stands in stark contrast to the comprehensive training they receive in finance, accounting, management, and marketing. New standard-setting organizations, like the Global Impact Investing Network and the International Foundation for Valuing Impacts, are seeking to fill the gaps in professional knowledge about impact. However, their theoretical grounding is weak and their guidance, especially regarding impact measurement, is widely considered inadequate. In some cases, they are adding to the confusion about what impact means and how it should be measured. I was motivated to provide training about impact by participants of my “scaling impact” workshop that is based on the book I coauthored (McLean & Gargani, 2019). The topic of impact is one that participants consistently find surprising and ask to learn more about. This workshop was created in response to their requests. The workshop was developed for professionals with an intermediate or advanced knowledge of evaluation, but it is suitable for novices and non-evaluators as well. Participants may come from any field, sector, or functional role. Social investors, program designers, managers, and evaluators.

McLean, R. & Gargani, J. (2019). Scaling impact: innovation for the public good. Routledge.
Suchman, E. (1968). Evaluative Research: Principles and Practice in Public Service and Social Action Programs. Russell Sage Foundation.
White, H. (2009). Some reflections on current debates in impact evaluation. New Delhi, India: International Initiative for Impact Evaluation.