Evaluation and Artificial Intelligence: Leveraging AI to Enhance Evaluation Quality

Abstract Information: Artificial intelligence (AI) will be revolutionizing the field of evaluation. AI increases the efficiency and effectiveness of our work as evaluators. Simultaneously, evaluators are informing and guiding the assessment of AI products and services. While AI presents opportunities to enhance productivity and quality, it also poses threats to equity, privacy, and practice. Evaluators need AI to advance the field yet we must use AI intelligently and responsibly in accordance with evaluation guiding principles and standards. At the same time, skilled evaluators are critical to understanding the risks and impacts of AI. The National Institute of Standards and Technology’s (NIST) response to the President’s 2023 Executive Order to “build U.S. capacity to evaluate and mitigate the risks of Artificial Intelligence” (www.nist.gov/artificial-intelligence) includes developing metrics and methods to evaluate AI as well as conducting evaluations of AI. In this session, David Fetterman and Sue Giancola will discuss how AI is transforming the field of evaluation from two perspectives: 1) enriching our own practice and 2) informing the public regarding the use and impact of AI. Generative AI can aid evaluators in conceptualizing evaluations, developing measures, analyzing data, and presenting results. In alignment with the theme of this year’s AEA conference, Giancola will examine the appropriate use of AI tools in these areas to advance diversity and cultural humility, reduce bias, and embrace multiple perspectives, as a means towards amplifying all voices in evaluation. Giancola will also discuss the sociocultural risks associated with using AI in evaluation and what evaluators can do to mitigate these risks. Fetterman will build on Giancola’s presentation by unpacking AI. He will describe how he built his empowerment evaluation GPT. De-constructing a GPT helps demystify AI. Understanding how AI functions is paramount to guiding the evaluation of AI applications and services. Fetterman will highlight his USAID and TIFA-funded work to develop guidelines for evaluating AI products and services in India. Together, Fetterman and Giancola will examine the ways in which evaluators can responsibly capitalize upon AI in their own work; develop tools to understand the effects of AI on people and processes; and assess the impact of AI in a global environment.