Inviting Artificial Intelligence Into the Undergraduate Evaluation Classroom

Abstract Information: As the fall 2023 episode of New Directions in Evaluation, focused on “Evaluation and Artificial Intelligence,” and standing-room-only sessions at the October 2024 AEA Conference demonstrated, the use of Artificial Intelligence (AI) in evaluation is a hot topic among evaluators today. While there is not yet consensus about whether, when, and how AI tools can or should be used in our work, the conversation is lively. As our field grapples with these questions, we need to develop strategies to share this topic with our students, who are potential future evaluators, commissioners, and users of evaluation. Despite her own discomfort with the disruptions associated with AI, the presenter has accepted that generative AI tools and their outputs are relevant to Evaluator Competencies 1.0: Professional Practice and 2.0 Methodology, and therefore need to be addressed with students. In this session, the presenter will describe how she brought AI tools into the program evaluation classroom and discuss their impact in shaping undergraduate students’ mastery of program evaluation competencies and artificial intelligence (AI) literacy, measured via the AI Literacy Scale (AILS) (Wang, Tau, & Yuan, 2022) and Program Evaluation Competency Scale (PECS-17) (Hou, 2022). This IRB-approved research, which employs a pre/post design, was conducted during the Spring 2024 semester. The presenter will share in detail, when, how, and which AI tools were brought into an evaluation classroom; how students were invited to use AI in their assigned work; and what resulted from these choices, as well as sharing the tools and assignments used. By doing so, she hopes to provide evidence about the utility of these strategies to support student learning and offer other instructors with similar goals concrete, easily-deployed strategies for teaching evaluation competency and AI Literacy.