130 - Incorporating unheard voices: Leveraging technology to expand evaluations

Abstract Information: Goal/

Objective: The goal of this roundtable is to introduce participants to new and existing technology to expedite and enrich their work. Tools aimed at automation and Artificial Intelligence (AI) have expanded greatly. Their responsible incorporation into one's research and evaluation repertoire can enhance the production and capacities of evaluators at all professional levels—from young and emerging to seasoned in the field. Breaking down barriers like time and effort will amplify often unheard voices and empower evaluators to access literature and methods with more depth and breadth. Using AI-supported search engines can expedite literature reviews and allow individuals to identify and access more literature, go deeper and seek out voices that are not seminal and widely known in the field. This would enhance evaluators' understanding of programmatic background, best practices and emerging methods and ideas. Access to these other sources and voices will enable independent consultants as well as individuals with corporate or academic backing alike to better contextualize evaluation findings and interpretation. As evaluators, we are often called upon for our methodological experience but not necessarily content expertise. More efficient access to information—and leveraging additional sources with fewer time constraints—will both improve our work but also increase our chances of acquiring new projects and branching into new fields. Developing knowledge and skills in data literacy and AI is critical to the responsible and appropriate involvement of AI in the field of evaluation. Our session will serve as a discussion amongst evaluators that will help to shape the adoption of such technologies.

Methods: During this roundtable, we will introduce tools widely available for little or no cost to enhance access to and assessment of information. We will also introduce considerations and best practices to ensure participants are able to responsibly leverage technology, including AI, and understand the sources and results. In addition to identifying and introducing specific tools, this will serve as a skill-building session and a discussion where participants will have the opportunity to identify potential uses in their work. Additionally, we will offer lessons learned from our exploration of technology in evaluation, sharing tips and tricks to make the process more efficient (e.g., prompt development and engineering, search operators). This open forum will allow individuals to truly see the potential applications in their work and to discuss how to best incorporate specific tools and technology more broadly. At the end, participants will understand how using technology (e.g., AI) can help them expand their work (e.g., conduct more comprehensive literature reviews) and make sure literature from broader audiences can be recognized and incorporated into their work. Finally, participants will see how technology can increase rigor through replicability and continuous updates.