How to get the buy-in from stakeholders to drive change: A case of reforming funding selection criteria and a case of automated application screening

Abstract Information: From an evaluator's perspective in an organisation, how do you showcase the value of changing the status quo of a process that runs smoothly, but where the quality of the outcome could be questioned? In this session, we share our experiences by way of an example of how we searched for openings in our organisation to get stakeholders (programme owners) on board and have them share our vision for improving the quality of their work and making them shine. Our case is about how we incidentally stumbled into developmental evaluation. We set out to evaluate gender bias in grant selection in a research collaboration with other international funders and researchers through the collaborative Research on Research Institute, RoRI on the CRITERIA project studying if grant application criteria influence inequalities in research funding. The project was presented at EVAL 2022. An epiphany in the project and feedback from Robin Wagner, NIH at the EVAL 2022 led us on a different mission to restructure the evaluation criteria for the selection of research grants in the Novo Nordisk Foundation. In this session, we present both how we went about restructuring our criteria set but we also share our experiences of trying to build a culture of learning through evaluation. This involves stakeholder relationships, being respectful and inclusive, yet persistent. Lastly, we will share our experiences with automated pre-screening of applications for a global program that generates an unusually high number of applicants, practically challenging our available evaluation capacity. This third and final presentation will show the approach in less technical terms and discuss considerations, including ethical considerations, to implement a non-human pre-selection mechanism and safeguards to avoid unfair treatment. Our endeavors in this direction follow our strategy to pursue AI solutions when relevant and appropriate.