Using AI to strengthen the evaluation of complex development programs

Abstract Information: The goal of this session is to demonstrate the value-added of AI tools and techniques in strengthening important areas of development evaluation. We have selected the evaluation of complex development programs for two reasons. First, while there is widespread recognition that many (if not most) development programs are “complex”, most current evaluation methodologies do not adequately address complexity. Second, one of the reasons that most evaluations have been slow to address complexity, is because many current evaluations often lack the tools and techniques for the collection and analysis of the many kinds of data required to model and evaluate complexity. AI, and its access to big data provides many of these tools – in increasingly accessible and affordable ways. Consequently, the evaluation of complexity provides a framework to demonstrate the areas of comparative advantage of AI and how they can be applied in practice. However, our message is that AI builds on, and complements current evaluation methodologies, but does not replace them. The panel will include two presentations and a combined discussant/presentation. The first presentation provides an introduction to complexity and presents a 5-step framework for the evaluation of complex development programs. This draws on the forthcoming publication Dealing with complexity in development evaluation (Bamberger and Zazueta). It also discusses some of the methodological challenges that have slowed the development and use of complexity-responsive evaluation methodologies and identifies some of the ways in which AI (and big data) can address these challenges. The second presentation will expand on how the comparative advantages of AI can be used to address the challenges facing the evaluation of complex programs. Two case studies will illustrate the use of precision causal modelling in the evaluation of complex development programs. The first case illustrates a bottom-up approach (Program to Aid Citizen Enterprise – PACE); while the second illustrates a top-down approach (Gemma Services Residential Behavioral Services for Children). Both cases emphasize the importance of a mixed-methods approach. The discussant will comment on the two presentations, and then describe a case from the ongoing work of the International Initiative for Impact Evaluation (3ie) that can complement the approaches discussed in the two presentations. The case will describe the use of machine learning to provide a broader framework for understanding the complex interactions (heterogeneity) among treatment effects, and will illustrate how the approach was applied to a previously completed randomized control trial (RCT) of a school-based gender attitude change program in Haryana, India.