Artificial Intelligence in Evaluation: A Comprehensive Review of Current Trends and Future Directions

Abstract Information: Despite the promising role of Artificial Intelligence (AI) in evaluation across diverse fields such as healthcare and education, the depth of its impact, ethical considerations, and practical applications remain insufficiently explored. This study aims to elucidate these aspects and propose future directions by systematically reviewing recent literature on AI in evaluation. Grasping AI's role in evaluation is crucial for maximizing its potential to improve decision-making, elevate program outcomes, and ensure fairness in AI-powered evaluation and related AI development. As AI technologies continue to advance, examining their implications in evaluation practices is vital for guiding research, policy formulation, and application strategies. Adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines, a detailed literature search was conducted across Google Scholar and IEEE Xplore using predefined search terms. The goal was to identify relevant studies on AI applications in evaluation, focusing on those published within the last ten years and peer-reviewed. Out of 5,510 records initially found, 31 were rigorously selected for comprehensive analysis after screening for relevance, timeliness, and scholarly credibility. The systematic review reveals a growing interest in deploying AI for evaluation purposes, notably within healthcare and education. Significant applications include AI’s contribution to causal inference (Linden & Yarnold, 2016), personalizing learning experiences (Zou et al., 2023), facilitating clinical decision-making (Kwong et al., 2023), and addressing equity in AI development (Reid, 2023). Challenges identified encompass ethical dilemmas, the necessity for interdisciplinary collaboration (Feuerriegel et al., 2020), and the importance of methodological transparency (Kwong et al., 2023). While underscoring AI's transformative capacity in evaluation practices, this study highlights AI’s potential to enhance the efficiency, accuracy, and fairness of evaluations and emphasizes the urgent need for ethical frameworks, policy guidelines, and further research. It proposes a socially just approach to leveraging AI's benefits while mitigating associated risks of marginalizing minoritized groups in evaluation. Additionally, this review outlines future directions, aiming to contribute to the development of ethical and culturally responsive, AI-driven evaluation practices across various sectors.

References: Feuerriegel, S., Dolata, M. & Schwabe, G. (2020) Fair AI. Business and Information System Engineering, 62, pp. 379–384 . https://doi.org/10.1007/s12599-020-00650-3 Kwong J.C.C, Khondker A., Lajkosz, K. (2023). APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support. JAMA Netw Open. 2023; 6(9):e2335377. https://doi.org/10.1001/jamanetworkopen.2023.35377 Linden, A., Yarnold, P. R. (2016). Combining machine learning and matching techniques to improve causal inference in program evaluation. Journal of Evaluation in Clinical Practice, 22(6), 868-874. https://doi.org/10.1111/jep.12592 Reid, A. M. (2023). Vision for an equitable AI world: The role of evaluation and evaluators to incite change. New Directions for Evaluation, 2023 (178-178), 111–121. https://doi.org/10.1002/ev.20559 Zou, B., Du, Y., Wang, Z., Chen, J., & Zhang, W. (2023). An Investigation Into Artificial Intelligence Speech Evaluation Programs With Automatic Feedback for Developing EFL Learners’ Speaking Skills. SAGE Open, 13(3). https://doi.org/10.1177/21582440231193818 Page M.J., Moher. D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C.D., et al. (2021) PRISMA explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ; 372: n160. doi: 10.1136/bmj.n160