Addressing Learning and Evaluation Challenges: New USAID Democracy, Human Rights, and Governance (DRG) Bureau Performance Evaluation Guidance

Abstract Information: Many evaluations and assessments involve a team of researchers conducting a large number of key informant interviews (KIIs), group interviews, and focus group discussions (FGDs) at one point in time over a three- to four-week period of fieldwork. Yet such studies regularly produce several complaints from commissioners, evaluators, and implementers related to the accuracy and reliability of the findings and the subsequent usefulness of the study. With partners Social Impact and The Cloudburst Group, the United States Agency for International Development’s Bureau for Democracy, Human Rights, and Governance (USAID/DRG) recently completed a study and guidance on addressing seven common challenges and pain points in qualitative evaluation work. These include 1) Case/site selection for small-n studies, 2) Selection of respondents, 3) Social desirability bias, 4) Qualitative data capture, 5) Qualitative data analysis, 6) Evidentiary support for statements, and 7) Clarity of findings to facilitate use. This guidance includes a) minimum standards to which all studies should adhere and b) good practices that studies should seek to implement when feasible. Through this inquiry, USAID/DRG also hopes to build consensus among USAID staff, learning partners (LPs), and evaluators on these minimum standards and best practices. While these studies focus on rapid qualitative PEs in the DRG sector, guidelines should be widely applicable across sectors. The study followed a sequenced approach, including (1) identifying team priors and building on the teams’ extensive experience, (2) reviewing and coding 67 recent DRG PE reports, (3) reviewing the academic and practitioner guidance literature, (4) conducting key informant interviews with evaluation commissioners, experts at key evaluator and research contracting firms, and past team leaders, and (5) convening workshops to review draft guidance and fill in remaining gaps. This panel will present the guidance and invite reactions from the larger evaluation community, stimulating discussion of these and other challenges in undertaking PEs, standards within the evaluator community, and emerging practices for circumventing these issues.