Tapestries of Impacts: Exploring the importance of impact scoping and context mapping in evaluation

Abstract Information: Impact evaluation plays a vital role in research and development (R&D), not only for gauging if an intervention is reaching its goals, but also for promoting accountability in resource allocation. Another value attached to it is its usefulness in deepening the understanding about how changes in well-being are attributable to the intervention, knowledge that is relevant to the policy process. Considering the growing realization that most R&D projects are usually complex, occurring in dynamic settings, and multidisciplinary in nature, more holistic and multidimensional approaches are gradually gaining traction in impact evaluation research. One holistic approach to impact evaluation is the use of exploratory sequential mixed methods impact assessment. It offers a flexible and iterative process of evaluation which begins with extensive qualitative data collection and analysis, which in turn informs quantitative data collection and analysis. As it begins with qualitative assessments, the discussion will highlight the value of qualitative aspects of this impact assessment methodology, specifically the context mapping and impact scoping phase. This phase examines both the project design review and the extent of the project’s impact. Impact scoping determines the magnitude to which the project has influenced various levels, such as personal, household, community, or institutional level. Context mapping provides a shared understanding about the broader context within which the project that is being evaluated sits. To illustrate the deployment and value of this in impact evaluation, this paper will discuss an ongoing evaluation of a revitalization program in textile industry in the Philippines. The program aims to improve the technological innovativeness of handloom weavers through provisioning of standardized handlooms and capacity building. Context mapping and impact scoping was done in two beneficiary weaver groups of different profile but identified to be both “ripe for technological challenges” by the program. Through this methodological step, terminal reports and project managers’ information were validated directly from the beneficiaries, and the same set of questions were deployed in two areas but resulted in opposite responses. Area 1 had zero knowledge of weaving prior to the program and showed a high interest and appreciation. While Area 2 already had weaving practices and production in place prior to the intervention showed resistance and lower appreciation to the intervention provided. It is notable that the impacts from these areas present complexities for the evaluators, and significant information might be overlooked if standardized surveys are employed. It also allows the evaluators to devise a tailored fit research instruments per case, if necessary, to better understand the program being assessed. The importance of this methodological step is not only for impact evaluation as it also showed implications to how interventions, especially those which are service oriented, should be conducted. Context mapping and impact scoping add value to intervention design and project’s impact pathway. This also signifies the importance of establishing a theory of change and impact pathway prior to program implementation, guiding the eventual conduct of impact assessment.