We have knowledge gaps on GenAI for Social and Behavior Change (SBC) programming: Let’s develop a research and evaluation agenda together!

Abstract Information: An emerging suite of Artificial Intelligence (AI) tools and approaches, including Generative AI (GenAI), Natural Language Processing (NLP), and Large Language Models (LLMs), has the potential to transform how organizations encourage social and behavior change (SBC) in critical areas such as sexual and reproductive health, vaccine uptake, and sharing of mis-and disinformation. In addition to changing how we communicate; Emerging AI is being used in efforts to improve monitoring and evaluation of SBC programming. For example, Gen AI chatbots can help with collecting data to monitor how well digital services are doing; NLP can be used to improve online audience segmentation and design and deliver messages for specific audiences; different kinds of AI can be integrated into qualitative data analysis to enable greater speed and efficiency and to augment the amount of data that can be coded, analyzed, and summarized to aid with evaluation. There is much to learn across the sector, however, regarding these novel approaches. We are working to develop a shared research agenda for the sector, including research on how evaluators can use these tools in their work; how emerging AI tools compare to one another and to traditional approaches; how AI tools can amplify and augment the work done by professional evaluators (while recognizing the value trained evaluators bring to the table); and what ethical, legal and quality control considerations need to be in place in order to confidently use these tools in ways that uphold human rights and other standards. Since 2021, iMedia and the MERL Tech Initiative have been convening and learning on the theme of evaluating digital SBC with Gates Foundation Partners and the wider MERL Tech community. Our goal is to bring diverse perspectives into the research agenda building process, including geographical, thematic, sectoral, age, and skill/experience level. So far we have facilitated sessions with the Natural Language Processing Community of Practice (NLP-CoP)’s SBC Working Group, the ICT4D Conference in Accra, the African Evaluation Association Conference in Kigali, the Global Digital Development Forum, the SBCC Summit in Morocco, the Global Digital Health Forum, and the European Evaluation Society Conference. At this session we will share what we have learned so far and involve participants in reviewing the emerging research agenda, identifying additional critical knowledge gaps; and discussing ways to collaboratively address them. Participants will leave the session with an updated understanding of the Emerging AI landscape for evaluation of SBCC and the critical questions that remain to be answered. Post-conference, we welcome participants to join the NLP-COP's SBC Working Group to continue working on these emerging issues, share case studies, and collaborate together to implement this co-created research agenda going forward. This session address aspects of Evaluation Foundations and Methodology, Program Development and Design, and International Evaluation, Diversity, and Specific Populations.